A Minimal Reporting Standard for Autonomous AI Agent Incidents

This paper addresses a real gap, and its observation that published failure rates use different event definitions and denominators is correct and useful. The writing is clear and the author is candid about limitations. However, I have fundamental concerns about the framework and its utility as a reporting standard.

Fundamental concerns

Frequency is not an attribute of an incident.

An incident report describes one event. Frequency describes a population of events over some amount of exposure. The person reporting an incident rarely holds the exposure data. Mature reporting systems in aviation and drug safety collect incident reports and exposure data separately, and analysts join them later. Under this proposal, nearly every operational incident would be reported as “not estimable,” and the standard would reduce to the severity label alone.

Frequency is a weak indicator of probability.

We ask how often incidents occur because risk is a function of severity and probability of occurrence. Frequency counts what was observed. Probability of occurrence is the chance that an event occurs over a defined exposure under stated conditions. For risk, the relevant conditions are usually operational. Frequency can inform that estimate, but it is only one input and often a poor one. A raw ratio carries no uncertainty, depends on detection, and reflects the conditions under which it was collected. The paper never states why frequency is wanted, so it never asks whether frequency is the right quantity to collect.

The framework is built on controlled test conditions.

A ratio of incidents to trials exists only where someone counts the trials. Three of the four frequency sources are evaluation campaigns. The one operational source is reported as incidents per day, which tracks reporting volume as much as system behavior. For events over time, mean time between events is the more useful quantity. It also requires more than a mean. To fit distributions, test for trends, and determine whether a system is improving or degrading, you need the individual intervals between events. This is solvable. If each incident report includes a timestamp for when the event occurred, rather than when it was reported, the intervals can be reconstructed. This is the kind of statistical analysis that incident reporting can support, and the proposed standard does not collect what it requires. We need severity and probability tracked in deployment, and this framework does not yet reach deployment.

The severity scale measures behavioral escalation, not harm.

The five levels describe how far an agent progressed toward loss of control. Level 2 is defined in terms of graders and rewards, which have no meaning in deployment. Consider two cases. An agent that ignores a constraint and deletes a production database by mistake sits near L1. An agent that leaves a sandbox and does nothing of consequence is L5. The scale is decoupled from consequence in both directions. It also has no clear place for prompt injection where the agent is the victim, cascading failures across agents, human misuse through agents, or privacy loss. Levels 3 and 4 require a judgment that behavior was “deliberate,” which is difficult to verify from public information. Because of this decoupling, the levels are best suited to tightly controlled test conditions. They may be a useful tool for developers tracking agent behavior during development. They are less useful for deployers, regulators, insurers, affected parties, and others whose interest in incident severity is an interest in harm.

The two dimensions are never demonstrated together.

The severity chart and the frequency table draw on largely different sources. Only one source, Anthropic’s three incidents, receives both a severity level and a frequency. Risk estimation requires a probability for each severity level. A single rate that lumps severity levels together cannot distinguish a system with many minor events from one with a few critical events. The paper names this goal in its closing sections but never produces a frequency by severity level for any dataset. The retrospective analysis shows that the comparability problem exists. It does not show that the proposal resolves it.

Suggestions for improvement

--Reframe the paper as a diagnosis of why current frequency claims are not comparable. -

--State that frequency is collected to estimate probability for risk, and evaluate which reportable data best supports that estimate.

--Replace the frequency field with what a reporter can supply, such as an event occurrence timestamp, system and version, permissions, duration of autonomous operation, and detection method.

--Separate severity of harm from behavioral progression, and include credible potential severity so near misses retain their value.

--Test the scale on operational incidents with multiple annotators, and report rates and intervals for each severity level on a single dataset.

This is a thoughtful start on a hard problem.

This project tackles an important problem in AI incident reporting, i.e., how to standardize reporting of key information about the importance of the incident, which is important to triaging incidents and determining whether specific incidents are generalizable. Specifically, the project proposes that incidents report use a severity scale (proposed in the project) and a frequency metric reporting the number of observed incidents as a fraction of the number of "comparable trials." The idea to focus on severity and frequency is sound, but the execution in this project is lacking.

Specific issues with this project include:

• Lack of validation of the severity index
• Use of a single severity index rather than, e.g., separate indices for different types of misalignment/control incidents
• Difficulty in interpreting the severity index: It's not obvious what should be done in response to the different levels of the index, or why the boundaries for each level are set where they are. It could be better to start by figuring out what types of responses are appropriate to what types of incidents, then determining severity based on that.
• Underspecification of frequency metric: Although the metric itself is sound, determining what constitutes a "comparable trial" is likely to be quite complex in practice, and the project provides no guidance for how to do this.
• Lack of consideration of stakeholders / use cases: While the project aims to provide a standard for incident reporting, it does not address the question of which stakeholders or reporting regimes would use this standard. For instance, the frequency metric seems only estimable by frontier model developers and not external actors such as METR. Identifying intended use cases would help project scoping and would help make the project more impactful.