The Agency Foundations Challenge

The Challenge is now in progress! Rewatch the keynote if you weren't there for the start and get free replicate.ai credits for your work. See existing resources for AI safety and reinforcement learning along with interpretability for topic 1 through 3. Happy research hacking!

Explore agency foundations research for the development & alignment of AI systems

Ever wondered how human agency, i.e. the capacity to (causally) control the world, will interact with increasingly powerful AI and future AGI systems which may also want to control the world? Do you question whether AGIs trained to focus on truthfulness or that are "intent aligned" are sufficiently safe? Us too!

Join us on four research tracks in this two week challenge that kicks off with a hackathon hosted with Alignment Jams. Submit your final projects at the end of the two weeks on this page.

We are developing an agency foundations paradigm to start researching agency in AI-human interactions - and are kicking off our work with a hackathon. We selected a few topics to start, such as figuring out how to algorithmically describe agency "preservation", mechanistically interpret how neural networks represent agents and their capacities, and describe challenges in the governance of agency-preserving AI systems. More information of our conceptual goals for this hackathon are provided here: https://www.agencyfoundations.ai/hackathon.

Start: Introductory talks - September 8th: 18:00-19:30 CET.

• End: Submission deadline - September 24th night (any timezone).
• ‍Location: Online/Remote
• ‍Topics: (1) mechanistic interpretability; (2) RL/IRL; (3) game theory; (4) conceptual/governance (see here for more details)
• ‍Prizes: US$10,000 ($2,500 in each category)
• ‍Format: online submissions.

More details about specific prizes, categories and additional information to be posted the 1st week of September.

Sign up below to be notified before the kickoff! Read up on the schedule, see instructions for how to participate, and inspiration on the agency foundations website.

Rules

You will participate in teams of 1-5 people and submit a project on the entry submission page (available when the hackathon starts). Each project consists of multiple parts: 1) The PDF report, 2) a maximum 10-minute video overview (optional), 3) title, summary, and descriptions.

You are allowed to think about your project and engage with the starter resources before the hackathon starts but your core research work should happen during the duration of the hackathon.

• Friday September 8, 16:00 UTC: Keynote talk by Catalin Mitelut inspire your projects and provide an introduction to the topic. Tim Franzmeyer will present his work on altruistic RL agents. Esben Kran will also give a short overview of the logistics.
• Saturday and Sunday 14:00 UTC: Project discussion sessions on the Discord server.
• Friday September 22nd 14:00 UTC: A discussion and short talk.
• Sunday September 24th night (all time zones): Submission deadline!