Imagine a world where the software that runs your favorite apps, secures your online transactions, and safeguards your digital life can be outsmarted and taken over using a cleverly disguised piece of code. This is not the plot of the latest cyber-thriller; it has actually been a reality for years. How this will change—positively or negatively—as artificial intelligence (AI) takes on an ever-increasing role in software development is one of the great uncertainties surrounding this brave new world.
In an era where artificial intelligence promises to revolutionize the way we live and work, the conversation about its security implications cannot be ignored. As we increasingly rely on artificial intelligence for tasks ranging from the mundane to the mission-critical, the question is no longer simply, “Can artificial intelligence increase cyber security?” (of course!), but also “Can AI get hacked?” (yes!), “Can AI be used hack?” (of course!) and “Will AI produce secure software?” (well…). This thought leadership article is about the latter. Cydrill (a secure coding training company) dives into the complex landscape of vulnerabilities produced by artificial intelligence, with a particular focus on GitHub Copilot, to emphasize the imperative of secure coding practice in protecting our digital future.
You can test your secure coding skills with this short write-up Self assessment.
The security paradox of artificial intelligence
Artificial intelligence’s leap from an academic curiosity to a cornerstone of modern innovation happened rather suddenly. Its applications span a breathtaking array of fields, offering solutions that were once the stuff of science fiction. However, this rapid progress and adoption has outstripped the development of adequate security measures, leaving both AI and AI systems vulnerable to a variety of sophisticated attacks. Already seen? The same things happened when software – as such – took over many areas of our lives…
At the heart of many AI systems is machine learning, a technology that relies on large data sets to “learn” and make decisions. Ironically, AI’s strength—its ability to process and generalize vast amounts of data—is also its Achilles’ heel. The starting point of “whatever we find on the Internet” may not be perfect training data; unfortunately, wisdom masses may not be enough in this case. Moreover, hackers, armed with the right tools and knowledge, can manipulate this data to trick the AI into making wrong decisions or taking malicious actions.
Co-pilot in the crosshairs
GitHub Copilot, powered by OpenAI’s Codex, is a testament to the potential of artificial intelligence in coding. It’s designed to improve productivity by suggesting code snippets and even entire blocks of code. However, numerous studies have highlighted the dangers of relying entirely on this technology. It has been proven that a significant portion of the code generated by Copilot may contain security flaws, including vulnerabilities to common attacks such as SQL injection and buffer overflows.
The principle “Garbage In, Garbage Out” (GIGO) is particularly relevant here. AI models, including Copilot, are trained on existing data, and like any large language model, most of this training is unsupervised. If this training data is flawed (which is quite possible given that it comes from open source projects or large Q&A sites like Stack Overflow), the output, including code suggestions, can inherit and propagate these flaws . In the early days of Copilot, a study found that approximately 40% of code samples produced by Copilot when asked to complete code based on samples from the CWE Top 25 were vulnerable, highlighting the GIGO principle and the need for increased security awareness. A more extensive study from 2023 (Is GitHub’s Copilot as bad as humans at introducing vulnerabilities into code?) had slightly better results, but still far from good: by removing the vulnerable line of code from a real-world example vulnerability and asking Copilot to complete it, it recreated the vulnerability about 1/3 of the time and fixed the vulnerability only about 1/4 of the time . In addition, it performed very poorly on vulnerabilities related to lack of input validation, producing vulnerable code every time. This highlights that generative AI is ill-equipped to deal with malicious input if ‘silver bullet’-like solutions to address vulnerabilities (e.g. prepared statements) are not available.
A path to secure AI-based software development
Addressing the security challenges posed by artificial intelligence and tools like Copilot requires a multifaceted approach:
- Understanding vulnerability: It is important to recognize that code generated by artificial intelligence can be subject to the same types of attacks as “traditionally” developed software.
- Improving secure coding practices: Developers must be trained to code securely, taking into account the nuances of AI-generated code. This includes not only identifying potential vulnerabilities, but also understanding the mechanisms through which AI suggests specific code snippets, in order to predict and effectively mitigate risks.
- Customizing the SDLC: It’s not just technology. Processes should also take into account the subtle changes that AI will bring. When it comes to Copilot, code development is usually the focus. But requirements, design, maintenance, testing, and operation can also benefit from large language models.
- Constant vigilance and improvement: AI systems – just like the tools they power – are constantly evolving. Keeping up with this evolution means staying informed about the latest security research, understanding new vulnerabilities, and updating existing security practices accordingly.
Navigating the integration of AI tools like GitHub Copilot into the software development process is risky and requires not only a change in mindset, but also the adoption of robust strategies and technical solutions to mitigate potential vulnerabilities. Here are some practical tips designed to help developers ensure that their use of Copilot and similar AI-driven tools increases productivity without compromising security.
Implement strict input validation!
Practical implementation: Defensive programming is always at the core of secure coding. When accepting code suggestions from Copilot, especially for functions that process user input, apply strict input validation measures. Define rules for user input, create a list of allowed characters and data formats, and ensure that inputs are validated before processing. You can also ask Copilot to do this for you; sometimes actually works well!
Manage addictions securely!
Practical implementation: Copilot can suggest adding dependencies to your project, and attackers can use this to implement supply chain attacks via “packet hallucination”. Before including any suggested libraries, manually check their security status by checking for known vulnerabilities in databases such as the National Vulnerability Database (NVD) or perform a Software Composition Analysis (SCA) with tools such as OWASP Dependency-Check or npm audit for Node.js projects. These tools can automatically monitor and manage dependency security.
Conduct regular security assessments!
Practical implementation: Regardless of the source of the code, be it AI-generated or hand-crafted, conduct regular code reviews and tests with a focus on security. Combine approaches. Test statically (SAST) and dynamically (DAST), perform software composition analysis (SCA). Perform manual testing and complement it with automation. But remember to put people before tools: no tool or artificial intelligence can replace natural (human) intelligence.
Be gradual!
Practical implementation: First, let Copilot write your comments or debug logs – it’s already pretty good at them. Any mistake in them will not affect the security of your code anyway. Then, once you’re familiar with how it works, you can gradually let it generate more and more code snippets for the actual functionality.
Always check out what Copilot has to offer!
Practical implementation: Never just blindly accept what Copilot suggests. Remember, you’re a pilot, that’s “just”. Co-pilot! You and Copilot together can be a very effective team, but you’re still in charge, so you need to know what the expected code is and what the outcome should look like.
Experiment!
Practical implementation: Try different things and prompts (in chat mode). Try asking Copilot to tweak the code if you’re not happy with what you get. Try to understand how the Copilot “thinks” in certain situations and understand its strengths and weaknesses. What’s more, Copilot keeps getting better over time – so keep experimenting!
Be informed and educated!
Practical implementation: Continuously educate yourself and your team about the latest security threats and best practices. Follow security blogs, attend webinars and workshops, and participate in forums dedicated to secure coding. Knowledge is a powerful tool for identifying and mitigating potential vulnerabilities in code, AI-generated or not.
Conclusion
The importance of secure coding practices has never been more important as we navigate the uncharted waters of AI-generated code. Tools like GitHub Copilot present significant opportunities for growth and improvement, but also special challenges when it comes to the security of your code. Only by understanding these risks can we successfully balance efficiency with security and protect our infrastructure and data. Along this journey, Cydrill remains committed to empowering developers with the knowledge and tools needed to build a more secure digital future.
Cydrill’s blended learning provides proactive and effective secure coding training for developers at Fortune 500 companies worldwide. Combining instructor-led training, e-learning, hands-on labs and gamification, Cydrill provides a new and effective approach to learning how to code securely.
Check out Cydrill’s secure coding courses.