Multi-Agent Security Research Sprint

Cooperation and collusion in AI systems

Join us for two days of research hacking on important questions! This time, we're teaming up with Oxford University's Christian Schroeder de Witt to explore how AI systems might collude and where cooperation can fail.

See the logistics information slides here and watch the keynote below:

Introduction

This hackathon focuses on concrete problems in multi-agent security in the age of autonomous and agentic systems. We are especially interested in projects that highlight ways to enhance the credibility and trust guarantees of agentic AI within the next 1-2 years. We encourage the use of an interdisciplinary toolbox, including economics, mechanism design, game theory, cryptography, and auction theory. We expect most research projects to focus on frontier AI systems.

Some general directions we'd like to explore include:

• Decentralized commitment devices (or, in general, formal contracts) for AI security and cooperation.
• Collusion among generative model agents, using cryptographic contracts.
• Simulation of financial markets (e.g., high-frequency trading, lending, market making) using generative agents. We aim to determine if contracts can help stabilize these AI agents. For instance, can individually selfish agents with contracts achieve better outcomes than innately pro-social agents (i.e., those with modified reward functions)?

For more inspiration, check out the "More Resources" tab. To see the schedule, please go to the "Schedule" tab.

Requirements

There are none! We invite people with diverse backgrounds to come along. We're always excited to welcome new perspectives into this fascinating field.

We especially welcome students exploring AI safety, cybersecurity professionals entering the field, AI safety researchers, ML researchers, and academics in the field of systemic security.

What is this hackathon?

The Apart Sprints are weekend-long challenges hosted by Apart to help you get exposure to real-world problems and develop object-level work that takes the field one step closer to secure artificial intelligence!

Read more about the project here.

Host a local group

If you are part of a local machine learning or AI safety group, you are very welcome to set up a local in-person site to work together with people on this hackathon! We will have several across the world and you can easily sign up under the “Hackathon locations” tab above.

You can use the resources at the bottom of that page to share on social media and set up an event space quickly and easily.

Watch the talk by Christian Schroeder de Witt below and read the articles shared under the Readings section.

Starter code!

Generative Agents Environment

This notebook gives you an easy entry to set up a generative agent environment.

Go to the notebook here

Experimenting with LLMs

This notebook provides an overview of how you can work with LLMs, such as Replicate usage and more.

Go to the notebook here

Readings

• Game Manipulators - the Strategic Implications of Binding Contracts: https://arxiv.org/abs/2311.10586
• Cooperative AI via Decentralized Commitment Devices (https://arxiv.org/abs/2311.07815)
• Formal Contracting and AI Social Dilemma: https://arxiv.org/abs/2208.10469
• Mediated Multi-agent Reinforcement Learning: https://arxiv.org/pdf/2306.08419.pdf
• Mitigating Generative Agent Social Dilemmas: https://social-dilemmas.github.io/
• I See You! Robust Measurement of Adversarial Behavior: https://openreview.net/attachment?id=0O5vbRAWol&name=pdf
• Stackelberg Attacks on Auctions and Transaction Fee Mechanisms: https://arxiv.org/abs/2305.02178
• Illusory Attacks: Detectability Matters in Adversarial Attacks on Sequential Decision-Makers: https://openreview.net/forum?id=F5dhGCdyYh
• Secrete Collusion among Generative AI Agents (to appear - please email cs@robots.ox.ac.uk for a copy)

Optional readings

• CredibleCommitments.WTF: https://hackmd.io/@sxysun/ccdwtf
• Why Crypto and X-Risk Researchers should listen to each other more: https://medium.com/@VitalikButerin/why-cryptoeconomics-and-x-risk-researchers-should-listen-to-each-other-more-a2db72b3e86b
• The promise and challenge of crypto+ai applications: https://vitalik.eth.limo/general/2024/01/30/cryptoai.html

Schedule

The schedule is written in PST (UTC-8). To get the schedule corresponding to your own time zone, please subscribe to the calendar or go to the full-screen view.

• Fri 11:00-12:00 - Keynote talk with Christian Schroeder de Witt and Esben Kran. Livestreamed and recorded.
• Fri 14:00-14:45 - Official online team matching for those who did not join with a team
• Sat 12:00-13:00 - HackTalk on commitment mechanisms with sxysun
• Sat 13:00-14:00 - HackTalk on multi-agent escalation with Juan Pablo
• Sat 15:00-16:00 - Office hours with established researchers - an amazing chance to get feedback from the same people who will judge your projects 😉
• Sun 10:00-11:00 - Another office hour online for those who have projects to discuss!
• Sun 20:00-21:00 - Join us for any questions regarding submission (you are of course welcome to share these in the Questions channel as well)
• Sun 22:00 - ⏰ Submission deadline!

See the full-screen view to change it to your time zone here: