AI Verification Grand Challenge
In two stages: several weeks online, then in-person finals in Singapore for the top teams.
Registration open
Overview
AI is getting more powerful very quickly, and some of the risks could be catastrophic. Companies are racing each other, so none of them can easily slow down alone.
One way out is for the US and China to agree to pace how fast the most advanced AI systems are built. The hard part is trust. Each side needs proof that the other is keeping its word, without handing over its secrets.
AI verification is the technology that makes that proof possible. It means tools that can check what an AI data center is actually doing, so a deal can be enforced without anyone giving up their intellectual property.
Right now, those tools don’t exist in a form both sides would trust. That makes this a technical problem more than a diplomatic one. Until it’s solved, an international agreement on AI can’t really be enforced.
How it works
- Sign up: Registration is open. Stage dates and full details are coming soon.
- Online stage: Teams of up to five tackle five focused verification problems and are scored on live leaderboards.
- Finals in Singapore: The top teams from the online stage meet in person in Singapore, alongside verification researchers.
Program overview
- Duration: Online stage, then finals in person
- Locations: Remote · Singapore
- Format: Two challenge stages · Online → In person
- Support: Compute · APIs · Community · In-person networking + travel · Prizes
Why join
- Win prizes. Top performers earn cash prizes, and the finalists’ work is recognized publicly.
- Shape the field, not just a leaderboard. Your findings go straight into the research. They inform future challenge cycles and become the foundation for the Grand Challenges that follow.
- Follow a clear path from newcomer to contributor. Start online with access to the community. Climb the public leaderboard and become a finalist. Then meet people in person in Singapore.
- Get recognized for what you build. Both the online and in-person stages end with an awards ceremony and a publicly visible leaderboard. You leave with concrete, verifiable wins for your portfolio.
- Meet the people working on this. The in-person finals put you in the same room as verification experts and other top researchers.
Pick a track and try to break it.
Team slots will be filled on a rolling basis.
Focus areas
Please note, tracks are subject to change and will be confirmed ahead of challenge launch.
| Adversarial Workload Classification | Disguise an AI training run as ordinary inference to fool a monitor that watches GPU activity. |
| Verified Inference Beyond Dense Models | Try to fool a cryptographic check into thinking an unapproved AI model is actually an approved one. |
| Fake an Evaluation Attestation | Pose as a dishonest model provider and forge the signed record that a safety evaluation ran on a model inside a secure hardware enclave. |
| Hide Compute from a Proof-of-Work Check | Sneak an unapproved workload onto GPUs while still passing a verifier that checks only the approved work is running. |
| Sabotage an Inspector Agent | Act as a facility operator and fool an AI inspector with doctored evidence. |
Who should join
Security researchers and red-teamers, cryptographers, ML and GPU engineers, hardware and confidential-computing specialists, and AI agent builders, plus strong students and early-career researchers from any of these fields.
You don’t need an AI safety background. If you like breaking systems or proving they can’t be broken, there’s a challenge for you. The challenges span a wide range of difficulty and specialisms, so there’s an entry point whether you’re new to the field or a specialist.
Teams of up to five are encouraged, so bring complementary skills.
Tip: The best teams will mix one or two people who know the technical area with someone who thinks like an attacker.
Useful skill sets
- GPU performance engineering and ML infrastructure (CUDA, profiling, distributed training, LLM inference serving)
- Adversarial ML, including evading classifiers and monitors
- Applied cryptography (zero-knowledge proofs, verifiable computation, proof systems)
- Mixture-of-experts and other non-dense model architectures
- Confidential computing and hardware security (trusted execution environments, remote attestation, firmware)
- Offensive security, red-teaming and CTF experience, including cloud security
- LLM agent building and LLM red-teaming (prompt injection, jailbreaks, agent tooling)
- Theoretical computer science or blockchain protocol research
Resources
Verification Strategy
These links give a high-level overview of the requirements for a mutually-verified AI agreement and will be useful context to then dig into technical research in any of the 5 tracks. We would recommend starting with the post from The Option Space and digging into the remaining links to see how the specific research areas the 5 challenges focus on fit into the larger picture.
- Edward Kembery and B.A. Bacigal, The Option Space: An Introduction to AI Verification. Read this first if verification is new to you; it introduces the vocabulary used in other sources.
- RAND, Verifying International Agreements on AI: Six Layers of Verification. Read to see where your track fits in a full verification regime and what it would mean for an agreement if that mechanism broke.
- Naci Cankaya (MIRI TGT), A System Overview for Near-Term, Low-Trust AI Compute Verification. Read to see how the mechanisms in Tracks 1, 2 and 4 would fit together in a real deployment, and why every track assumes a prover who cheats only when unlikely to be caught.
- Scher and Thiergart (MIRI), Mechanisms to Verify International Agreements About AI Development. Use as a reference for which assumptions each mechanism depends on, since that assumption is usually what your attack should target.
- Amodo Design, AI 2040 Plan A: Verification SITREP. Read to see which problems practitioners consider unsolved, so you can show how your results bear on them.
- Harack et al. (Oxford AIGI), Verification for International AI Governance. Read if you want to frame your findings for a policy audience.
- Romeo Dean, AI 2040: Verification Supplement. Read to judge how soon an agreement would need the mechanism you are attacking.
- Yonadav Shavit, What Does it Take to Catch a Chinchilla? Read for the origin of training verification by spot-checking, which still shapes how the Track 1 harness checks your training run.
Track 1: Adversarial Workload Classification
Rahman & Tajdari’s work is what this track is based on and is essential for understanding the types of evasion strategies that can be employed against the workload classifier. Reading the additional sources will also help with designing adversarial attacks beyond L3.
Core Readings:
- Rahman and Tajdari (MIRI/UVA), Detecting Hidden ML Training With Zero-Overhead Telemetry. Read first: it describes the classifier you attack at L1 and L2, and the evasion strategies it was already hardened against, so you don't spend submissions repeating them.
- MIRI TGT, De-risking Interconnect Limits for AI Verification (code). Read before L3 to understand why multi-node training stands out in network traffic and how much an attack has to hide.
- Seferis and Fist, Detecting Compute Structuring in AI Governance Is Likely Feasible. Read to understand which traffic features the L3 and L4 monitor relies on.
Additional Resources:
- Poynting (LessWrong), Whitebox evasion is the baseline for GPU workload verification. A short read on why you should assume full knowledge of the monitor, and which weaknesses in GPU telemetry remain open.
- Douillard et al., DiLoCo, and Jaghouar et al., OpenDiLoCo. Read for working methods to make nodes communicate rarely, the most obvious route past L3.
- Gargiulo and Kulp, Workload Identification with Physical Side Channels (dataset). Read to anticipate what power monitoring adds at L4, and use the open traces to test disguises offline.
- NVIDIA, NVML API reference. Use as a reference when designing reshape or mimic attacks, since you need to know exactly what each counter measures.
- Samar Ansari, Hardware-Level Governance of AI Compute. Read for a feasibility rating of 20 hardware-level governance mechanisms, including which telemetry-based checks are mature enough to rely on.
- Jia et al., Proof-of-Learning: Definitions and Practice. Read to understand how the harness confirms you actually trained, so your attack keeps passing that check.
- MLSEC 2020 results. Read for how an attacker-versus-defender classifier competition ran and which evasion approaches worked.
Track 2: Forge an Inference Proof
Levels L1, L2, and L3 are a statistics problem (how far can M' drift from M while staying under VerInf's surprisal threshold T); L4 is a soundness problem (find a bug in an exact proof). Read VerInf's flaw log and the Hollow-LLM and zkml-audit papers first to get an understanding of the system being used.
Core:
- James Petrie (FLI), VerInf. Read first: it is the L1 to L3 target, and you need to understand its surprisal score and threshold to know how far M' can drift from M.
- Levi Neuwirth, Verified Inference Between Adversaries. Read to see which kinds of flaws have already been found and fixed in VerInf, which hints at where others may remain.
- Gong, Liu and Li, Hollow-LLM Attack. Read for a worked example of passing a valid proof while running a different model, the kind of result that wins this track.
- SASH, Towards Proving Frontier-Scale LLM Inference and The Math Behind Pacing the Frontier. Read before L4 to understand how the exact, deterministic proof differs from VerInf and what it assumes.
Additional Resources:
- Netzer Epstein et al., zkml-audit-benchmark. Read to learn the bug patterns most common in ZK-ML code, a starting checklist for auditing the L4 prover. Access requires approval on Hugging Face, so request it early.
- 0xPARC, zk-bug-tracker. Use as a reference of real ZK bug classes to check the L4 implementation against.
- Gao, Liang and Guestrin, Model Equality Testing: Which Model Is This API Serving? Read for tools to measure how distinguishable M' is from M, which tells you how much behaviour change you can afford.
- Karvonen, Reuter, Rinberg et al., DiFR: Inference Verification Despite Nondeterminism. Read to calibrate how small a model change can still be detected, a useful benchmark for your wrapper or M'.
- Sun, Li and Zhang, zkLLM. Read to understand the standard construction most ZK-LLM provers follow.
- Jim Miller (Trail of Bits), Frozen Heart. Read for a classic implementation flaw that let attackers forge proofs.
- Khovratovich, Rothblum and Soukhanov, How to Prove False Statements. Read if you want to attack the proof protocol itself rather than its code.
- Pearl Polymath Problem 7. Read for a related open problem on breaking proof-of-inference schemes.
Track 3: Fake an Evaluation Attestation
Each level removes one check from the full prototype, so finding effective adversarial strategies will likely require reading the documentation line by line and figuring out where there are gaps to exploit. Read Tinfoil's PROTOCOL.md and RECEIPT.md before anything else to understand how it works. Then, read the GPU confidential-computing security paper for L3, where the gap is the boot-time link between CPU and GPU attestation.
Core:
- Tinfoil, double-blind-eval: PROTOCOL.md and RECEIPT.md. Read first: this is target 1, and each level's gap is something these two documents do or don't bind together.
- OpenMined, Secure Enclaves for AI Evaluation (syft-enclave code). Read to understand target 2 and why letting approved job code do the scoring changes what you can attack.
- Tinfoil, Attestation architecture. Read to understand how a verifier checks which code an enclave runs, which you need in order to exploit the L2 gap.
Additional Resources:
- Gu et al. (IBM Research, Ohio State), Blueprint, Bootstrap, and Bridge: A Security Look at NVIDIA GPU Confidential Computing. Read before L3 to understand how CPU and GPU attestation are linked at boot, the link L3 leaves unchecked.
- OpenMined, Double-blind evaluation of a frontier model. Read to see how a real double-blind evaluation ran, which helps judge which attacks would matter in practice.
- Lucid Computing, Red-teaming at Lucid Labs. Read to see how a team building this kind of system plans to red-team it, with a government red team playing an operator who covertly swaps models.
- Schnabl et al., Attestable Audits. Read for an alternative design to compare against the targets, which can help you spot what they leave unbound.
- RAND, Highly Secure Inference Data Centers. Read for the security bar a fully hardened facility would set, as a yardstick for the targets.
Track 4: Hide Compute from a Proof-of-Work Check
The track asks how much of a chip's capacity a proof of useful work actually accounts for. Read the Pearl whitepaper's list of blocked shortcuts first (L3 removes one of them), then Basu's usefulness gap (which is tested in L2), then the kernel-efficiency material, since a faster kernel than the verifier's reference is itself covert capacity.
Core:
- Pearl Research Labs, Pearl FP8 whitepaper (code). Read first: you need the hardness assumption and the list of shortcuts the checks block, since L3 removes one of those checks.
- Abhinaba Basu, The Usefulness Gap in Proof-of-Useful-Work. Read before L2 to see why the counted work need not be useful, which is the opening L2 leaves.
- Pearl Polymath problems. Read for the problems the Pearl team itself considers open; #1 (security of the noise scheme) and #4 (training through cuPOW) overlap with this track.
Additional Resources:
- Komargodski and Weinstein, Proofs of Useful Work from Arbitrary Matrix Multiplication. Read if you want to attack the scheme's security argument rather than its implementation.
- Pearl PR #311. Read to see the FP8 mining implementation, which is not yet merged, and the open review comments on it.
- Shah, Dao et al., FlashAttention-3. Read to see how far kernel engineering can push GPU utilisation, since capacity the verifier's baseline misses is covert capacity for you.
- Badash, Boneh, Komargodski and Srivastava, Hawkeye. Read to understand how a verifier can recompute GPU results exactly, which constrains how you could forge tiles.
Track 5: Sabotage an Inspector Agent
A doctored evidence bundle is an indirect prompt injection against an LLM judge that returns one bit, so the most useful prior work is research on prompt injection and on attacking LLM judges. Read the Verifier Challenge write-up first for the 19 rules and the 15 evasions its automated red team found; Tier 2 (L3 and up) then needs forged telemetry that stays physically consistent.
Core:
- Jason Hausenloy and Jasmine Li, Inspector Agents prototype. Read this first: the code shows exactly which checks the inspector runs.
- Jeff Wu, Auditing the Singularity. Read for the general idea behind inspector agents, an AI auditor with full access to a tamper-evident audit trail, and what it is supposed to guarantee, so you know which promise your attack breaks.
- Hausenloy and Li, Verifier Challenge and traces dataset. Read the write-up for the 60-round automated red team, the 15 evasions it found and the 19 rules that resulted. Use the traces to see what real training and serving telemetry look like; their adversarial phases change only the labels, not the telemetry.
Additional Resources:
- Greshake et al., Not What You've Signed Up For: Indirect Prompt Injection. Read for the core attack pattern of this track: instructions hidden in the data an LLM is asked to judge.
- Raina, Liusie and Gales, Is LLM-as-a-Judge Robust? Read for methods to find one edit that fools an LLM judge across many inputs, which is what your attack(bundle) function needs to do.
- Motwani et al., Secret Collusion among AI Agents. Read if you take on the leaky-inspector extension, for how secrets can be hidden in permitted outputs.
- Rando, Debenedetti, Paleka and Tramèr (SPY Lab), Our competitions at IEEE SaTML 2024. Read for lessons from two competitions on attacking LLM defences, including why only adaptive, multi-turn attacks evaluated them properly.
FAQ
Do I need an AI safety background?
No. What matters is skill in an area the tracks cover, such as security, cryptography, ML infrastructure, hardware or LLM agents.
What is the application process?
Sign up with the form on this page. Candidates can choose their own challenge track. You can also choose to compete in more than one challenge track.
Full challenge rules coming soon.