Demonstrating LLM Code Injection Via Compromised Agent Tool
Kevin Vegda, Oliver Chamberlain, William Baird
Submitted to AI capabilities and risks demo-jam. Sprint projects are early-stage work by participants, not Apart Research publications.
This project demonstrates the vulnerability of AI-generated code to injection attacks by using a compromised multi-agent tool that generates Svelte code. The tool shows how malicious code can be injected during the code generation process, leading to the exfiltration of sensitive user information such as login credentials. This demo highlights the importance of robust security measures in AI-assisted development environments.
Reviews
This super cool, and seems to work pretty consistently! I could see this plausibly happening in the real world, especially when the code being generated is large enough to hide the attack. The Stackoverflow example you gave is realistic.I really like that you included an actual render of the UI, it makes it seem much more plausible.It would be a nice improvement for the injected code to do something (like hit an endpoint), rather than print a log. Maybe a bit tricky to get that working on hosted Gradio though.
Very good entry, I especially like the stackoverflow post.This worked so great that I thought the generating code had malfunctionned and failed to introduce vulnerabilities. The only caveat that I see is that it is unclear to me what vulnerability this demonstrates, as releasing a new AI tool to introduce vulnerabilities seems costly and like this would easily get shutdown.
Nice work! I like the StackOverflow entry point, and including the code only when the “copy” button is pressed! To make this more visceral, I’d aim to reduce the number of steps to see the reveal, and think about making the results of the demo clearer inside the experience.
Cite this project
@misc{vegda2024demonstrating,
title = {{Demonstrating LLM Code Injection Via Compromised Agent Tool}},
author = {Kevin Vegda and Oliver Chamberlain and William Baird},
year = {2024},
month = aug,
note = {Submitted to AI capabilities and risks demo-jam, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/demonstrating-llm-code-injection-via-compromised-agent-tool}},
url = {https://apartresearch.com/sprints/projects/demonstrating-llm-code-injection-via-compromised-agent-tool}
}More from AI capabilities and risks demo-jam
- 1st place by peer reviewView project: Speculative Consequences of A.I. Misuse
Speculative Consequences of A.I. Misuse
Team S.C.A.M.
This project uses A.I. Technology to spoof an influential online figure, Mr Beast, and use him to promote a fake scam website we created.
- View project: Phish Tycoon: phishing using voice cloning
Phish Tycoon: phishing using voice cloning
Phish Tycoon
This project is a public service announcement highlighting the risks of voice cloning, an AI technology capable of creating synthetic voices nearly indistinguishable from real ones. The demo involves recording a user's …
- View project: Misinformational AI-Generated Academic Papers
Misinformational AI-Generated Academic Papers
The Fake Academics
This study explores the potential for generative AI to produce convincing fake research papers, highlighting the growing threat of AI-generated misinformation. We demonstrate a semi-automated pipeline using large …