Aug 26, 2024
CoPirate
Mia Hopman, Carissa Cullen, Jack Wittmayer, Vaishnavi Pamulapati
As the capabilities of Artificial Intelligence (AI) systems continue to rapidly progress, the security risks of using them for seemingly minor tasks can have significant consequences. The primary objective of our demo is to showcase this duality in capabilities: its ability to assist in completing a programming task, such as developing a Tic-Tac-Toe game, and its potential to exploit system vulnerabilities by inserting malicious code to gain access to a user's files.
Lucas Hansen
Very cool that you built out a full UI of a specific example. The in-app explanation of what happened is also quite helpful. I think it would’ve been a bit more impactful if the malicious code was less obvious and accomplished a desirable (to the hacker), rather than just destructive, task.
Épiphanie Gédéon
Great entry!You really can feel the rush and “there’s no time to look at everything”, and the copy paste. The demo does a good job of conveying a sense of overwhelm to the player and when they might rely on coding assistants too much.It does need some polish overall, especially with the document visualization, and I also feel like it would have been better if it used a fake pypi repo that had been “compromised” instead of direct system prompts, but the way the model disguises the removal of the files is very good as it is.The subject matter is important, in that we might head toward more and more unsupervised automation which means more point for vulnerabilities.To me, this is a solid entry and would love to see a polished version of it.
Adam Binksmith
A nicely executed project! The simulation → debrief format is well executed, and the task is short enough that many users would make it to debrief. A next step might be to think more about the threat model shown - in what situations would targets be using a compromised coding assistant, and is there a more direct threat model in those cases?
Cite this work
@misc {
title={
@misc {
},
author={
Mia Hopman, Carissa Cullen, Jack Wittmayer, Vaishnavi Pamulapati
},
date={
8/26/24
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
