What Breaks Next
Azka Qadir
Submitted to AI Incident Response Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.
This project combines an accident taxonomy with a practical one day verification protocol for agentic AI evaluations. I identify twelve recurring accident classes from recent incidents and research, then translate them into sixteen checks that labs and defenders can use to test containment, authority, monitoring and evidence quality before or after high capability evaluations.
Reviews
A clear, well-organized project: twelve recurring AI-incident failure patterns turned into sixteen practical one-day checks. Genuinely useful as a checklist, and grounded in real, cited incidents. The main gap, which the author states plainly, is that none of the checks have been tried out yet - a strong blueprint, not yet a tested result. Running even a few checks against a small setup next would go a long way.
This project tackles the issue of improper controls by testing them and scoping them. As seen in various evaluations done by third parties, appropriate access controls were not integrated on their end causing cyber attacks during evaluation runs. Testing is a good way to check, it would be better if these could be verified using something such as remote attestations to ensure third parties have a correctly configured sandbox.
Cite this project
@misc{qadir2026breaks,
title = {{What Breaks Next}},
author = {Azka Qadir},
year = {2026},
month = sep,
note = {Submitted to AI Incident Response Sprint, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/what-breaks-next-4aza}},
url = {https://apartresearch.com/sprints/projects/what-breaks-next-4aza}
}More from AI Incident Response Sprint
- 1st placeView project: Noticing the escapes: Who spots a runaway agent, and how long does it take?
Noticing the escapes: Who spots a runaway agent, and how long does it take?
The Escapees
In 2026, nine incidents became public in which AI agents escaped their environment and did something unauthorized to other parties. We analyzed all of these incidents and how they were noticed initially. None of these …
- 2nd placeView project: Strengthening International AI Incident Discovery
Strengthening International AI Incident Discovery
Team A-O-I
Mandatory AI incident reporting exists to make dangerous events visible to regulators while there is still time to act. We coded the six such regimes in force or set to be enforced soon from primary legal texts and …
- 3rd placeView project: When to Ask: An RL Environment That Teaches AI Agents to Act on What Users Mean and Not Just What They Say
When to Ask: An RL Environment That Teaches AI Agents to Act on What Users Mean and Not Just What They Say
Teachafy
When to Ask is a reinforcement-learning environment that trains AI agents to work out what the user actually means before they use a powerful credential, instead of just carrying out the literal instruction. Agents …