Skip to content
Sprint projectAug 26, 2024

CoPirate

Mia Hopman, Carissa Cullen, Jack Wittmayer, Vaishnavi Pamulapati · Team CoPirates

Submitted to AI capabilities and risks demo-jam. Sprint projects are early-stage work by participants, not Apart Research publications.

As the capabilities of Artificial Intelligence (AI) systems continue to rapidly progress, the security risks of using them for seemingly minor tasks can have significant consequences. The primary objective of our demo is to showcase this duality in capabilities: its ability to assist in completing a programming task, such as developing a Tic-Tac-Toe game, and its potential to exploit system vulnerabilities by inserting malicious code to gain access to a user's files.

Reviews

Judging this Sprint?

Review this project

Your public critique appears on this page without your name. Your private critique is not published; only the Apart team reads it. If you agree below, we share your review with grantmaking.ai (opens in new tab) and the Transformative AI Fund so strong projects can be funded.

Not shown on this page.

Shown on this page, without your name.

Only the Apart team reads this, and funders if you agree below.

Share my name publicly on grantmaking.ai *
Share my private critique with funders *

  1. Very cool that you built out a full UI of a specific example. The in-app explanation of what happened is also quite helpful. I think it would’ve been a bit more impactful if the malicious code was less obvious and accomplished a desirable (to the hacker), rather than just destructive, task.

  2. Great entry!You really can feel the rush and “there’s no time to look at everything”, and the copy paste. The demo does a good job of conveying a sense of overwhelm to the player and when they might rely on coding assistants too much.It does need some polish overall, especially with the document visualization, and I also feel like it would have been better if it used a fake pypi repo that had been “compromised” instead of direct system prompts, but the way the model disguises the removal of the files is very good as it is.The subject matter is important, in that we might head toward more and more unsupervised automation which means more point for vulnerabilities.To me, this is a solid entry and would love to see a polished version of it.

  3. A nicely executed project! The simulation → debrief format is well executed, and the task is short enough that many users would make it to debrief. A next step might be to think more about the threat model shown - in what situations would targets be using a compromised coding assistant, and is there a more direct threat model in those cases?

Cite this project

@misc{hopman2024copirate,
  title = {{CoPirate}},
  author = {Mia Hopman and Carissa Cullen and Jack Wittmayer and Vaishnavi Pamulapati},
  year = {2024},
  month = aug,
  note = {Submitted to AI capabilities and risks demo-jam, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/copirate}},
  url = {https://apartresearch.com/sprints/projects/copirate}
}

Build something like this at the next Sprint

AI Collusion Research Sprint · Oct 23 - 25, 2026