CoPirate
Mia Hopman, Carissa Cullen, Jack Wittmayer, Vaishnavi Pamulapati · Team CoPirates
Submitted to AI capabilities and risks demo-jam. Sprint projects are early-stage work by participants, not Apart Research publications.
As the capabilities of Artificial Intelligence (AI) systems continue to rapidly progress, the security risks of using them for seemingly minor tasks can have significant consequences. The primary objective of our demo is to showcase this duality in capabilities: its ability to assist in completing a programming task, such as developing a Tic-Tac-Toe game, and its potential to exploit system vulnerabilities by inserting malicious code to gain access to a user's files.
Reviews
Very cool that you built out a full UI of a specific example. The in-app explanation of what happened is also quite helpful. I think it would’ve been a bit more impactful if the malicious code was less obvious and accomplished a desirable (to the hacker), rather than just destructive, task.
Great entry!You really can feel the rush and “there’s no time to look at everything”, and the copy paste. The demo does a good job of conveying a sense of overwhelm to the player and when they might rely on coding assistants too much.It does need some polish overall, especially with the document visualization, and I also feel like it would have been better if it used a fake pypi repo that had been “compromised” instead of direct system prompts, but the way the model disguises the removal of the files is very good as it is.The subject matter is important, in that we might head toward more and more unsupervised automation which means more point for vulnerabilities.To me, this is a solid entry and would love to see a polished version of it.
A nicely executed project! The simulation → debrief format is well executed, and the task is short enough that many users would make it to debrief. A next step might be to think more about the threat model shown - in what situations would targets be using a compromised coding assistant, and is there a more direct threat model in those cases?
Cite this project
@misc{hopman2024copirate,
title = {{CoPirate}},
author = {Mia Hopman and Carissa Cullen and Jack Wittmayer and Vaishnavi Pamulapati},
year = {2024},
month = aug,
note = {Submitted to AI capabilities and risks demo-jam, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/copirate}},
url = {https://apartresearch.com/sprints/projects/copirate}
}More from AI capabilities and risks demo-jam
- 1st place by peer reviewView project: Speculative Consequences of A.I. Misuse
Speculative Consequences of A.I. Misuse
Team S.C.A.M.
This project uses A.I. Technology to spoof an influential online figure, Mr Beast, and use him to promote a fake scam website we created.
- View project: Demonstrating LLM Code Injection Via Compromised Agent Tool
Demonstrating LLM Code Injection Via Compromised Agent Tool
This project demonstrates the vulnerability of AI-generated code to injection attacks by using a compromised multi-agent tool that generates Svelte code. The tool shows how malicious code can be injected during the code …
- View project: Phish Tycoon: phishing using voice cloning
Phish Tycoon: phishing using voice cloning
Phish Tycoon
This project is a public service announcement highlighting the risks of voice cloning, an AI technology capable of creating synthetic voices nearly indistinguishable from real ones. The demo involves recording a user's …