Red Teaming A Narrow Path: ControlAI Policy Sprint

This Sprint has been concluded and we have an amazing set of winners

• Prize 1: Four Paths to Failure: Red Teaming ASI Governance: https://apartresearch.com/project/four-paths-to-failure-red-teaming-asi-governance-se53,
• Prize 2: Red Teaming A Narrow Path: ControlAI Policy Sprint by Aritra Das and Vaani Goenka: https://apartresearch.com/project/red-teaming-a-narrow-path-controlai-policy-sprint-by-aritra-das-and-vaani-goenka-w5w9,

Join us for a critical one-day sprint to red team the policy framework that could determine humanity's future relationship with artificial intelligence. ControlAI has developed "A Narrow Path" - the first comprehensive plan to address extinction risks from Artificial Superintelligence (ASI). These policies are already being actively pushed to lawmakers in the UK and US through their Direct Institutional Plan (DIP), making this red teaming exercise directly relevant to real-world policy implementation.

Your mission: Help strengthen the Phase 0 policies by identifying weaknesses, implementation challenges, and gaps before they reach legislators' desks.

The Challenge We Face

While most AI developments are beneficial, the rise of superintelligent AI (ASI) threatens humanity with extinction. We do not know how to control AI vastly more powerful than us. Should attempts to build superintelligence succeed, this would risk our extinction as a species.

Current AI development is proceeding without adequate safety measures, with reasonable estimates indicating that it could cost only tens to hundreds of billions of dollars to create artificial superintelligence. Meanwhile, the very people developing advanced AI are warning about these risks.

A Narrow Path: The Solution

ControlAI has found no other plan that comprehensively tries to address the issue, so they made one. "A Narrow Path" is structured in three phases:

• Phase 0: Safety - Immediate policies to prevent ASI development for 20 years
• Phase 1: Stability - International oversight that doesn't collapse over time
• Phase 2: Flourishing - Building foundations for safe transformative AI under human control

Real-World Impact

This isn't theoretical policy research. ControlAI launched a pilot campaign focused on UK lawmakers that validated their approach. In less than three months, over 20 cross-party UK parliamentarians publicly supported their campaign. They succeeded in gaining support in 1 out of every 3 cases when briefing lawmakers.

Recent polling shows that a large majority (74%) of Brits support placing the UK's AI Safety Institute on a statutory footing, and 16 British lawmakers have signed a statement calling for new AI laws targeted specifically at "superintelligent" AI systems.

Focus of This Sprint

We're exclusively red teaming Phase 0 policies because:

1. These are the policies actively being pushed through the DIP
2. They are the necessary first step - stopping ASI development is the precondition for everything else
3. Real-world implementation is imminent, making your feedback immediately actionable

Prizes

1st Place: $300 + exclusive mentoring session with Control AI's policy team to discuss your findings and potential integration into their Direct Institutional Plan advocacy efforts.

2nd Place: $200 + mentoring session with Control AI's policy team to explore how your red teaming insights could strengthen real-world policy implementation.

Primary Document

• A Narrow Path - Full Document - The complete policy framework
• Phase 0 Summary - Condensed version focusing on immediate safety policies (provided in sprint materials)

Context Materials

• ControlAI Website - Organization background and current campaigns
• Direct Institutional Plan - How these policies are being implemented in practice
• Control AI Recent Advocacy Results - Real-world traction with policymakers

Phase 0 Policy Overview

Goal: Ensure no one builds ASI (superhuman + general AI) for at least 20 years.

Core Policies:

1. Prohibitions:
   • Ban on building ASI itself
   • Ban on precursor capabilities (automated AI research, advanced hacking)
2. Enforcement Mechanisms:
   • Safety cases required before training (proof the AI won't become ASI)
   • Three-tier licensing system:
     • Training license (>10²⁵ FLOP training runs)
     • Compute license (>10¹⁷ FLOP/s providers)
     • Application license (using licensed systems)
3. International Coordination:
   • Global treaty harmonizing prohibitions and regulations

Your submission must include:

1. A completed project report using the provided template :Make your own copy of the template on Google Docs
2. Link to a public GitHub repository with your analysis code (optional but recommended)
3. A brief (2-3 minute) video presentation (optional but recommended)

Important: Include LLM Usage Documentation

All submissions must include an appendix called "LLM Prompts Used" that documents any AI assistance used in your project. This includes prompts for:

• Policy research and background analysis
• Historical precedent identification and comparison
• Evidence gathering and fact-checking assistance
• Red teaming methodology development
• Report writing and editing assistance

This transparency helps others understand your methodology

1. Implementation Feasibility Analysis (33.3%)

• Does the analysis realistically assess what government agencies, resources, and expertise would be needed to implement these policies?
• Are the identified implementation challenges specific and grounded in understanding of how similar policies have worked (or failed) in practice?
• Does the submission adequately consider bureaucratic, technical, and coordination complexities involved in enforcement?
• How well does the analysis account for real-world constraints like budget limitations, regulatory capture, and inter-agency coordination?

2. Policy Effectiveness Assessment (33.3%)

• Does the analysis identify specific ways the policies could fail to prevent ASI development or be circumvented by determined actors?
• How thoroughly does the submission examine edge cases, loopholes, or unintended consequences that could undermine the 20-year goal?
• Does the assessment consider different threat models (state actors, rogue researchers, corporate actors) and how policies address each?
• Are the identified failure modes realistic and significant, or primarily theoretical edge cases?

3. Evidence-Based Reasoning (33.3%)

• Does the submission cite relevant historical examples of similar policies (nuclear non-proliferation, export controls, dual-use technology regulation) to support its arguments?
• Are claims backed by empirical data, documented case studies, or credible expert analysis rather than speculation?
• How well does the analysis draw lessons from comparable regulatory domains to assess likely outcomes?
• Does the submission avoid making unsupported assertions about what "would" or "could" happen without evidence?

⏰ Schedule on Discord

All times UTC

09:00 - 09:45 | Opening & Context Setting

• 09:00-09:30: Welcome Keynote by Andrea Miotti
• 09:30-09:45: Q&A and team formation

09:45 - 22:00 | Red Teaming Sprint

• 13:00 UTC Office Hour:https://discord.gg/5WFqeVZn9c?event=1380517711649706085
• 21: 00 UTC Office Hour: https://discord.gg/5WFqeVZn9c?event=1380518441320059021

22:00-24:00 |Wrap-up

• Put Finals Touches to Hackathon Project
• Submit on the website before 12am UTC

Post-Event

• By June 21: Reviews Wrap Up for Hackathon
• By June 22: Winners Announced and Short Presentation by Winners
• Follow-up: Selected insights may be shared with ControlAI's policy team and incorporated into DIP advocacy