A Minimal Reporting Standard for Autonomous AI Agent Incidents

This paper addresses a real gap, and its observation that published failure rates use different event definitions and denominators is correct and useful. The writing is clear and the author is candid about limitations. However, I have fundamental concerns about the framework and its utility as a reporting standard.

Fundamental concerns

Frequency is not an attribute of an incident.

An incident report describes one event. Frequency describes a population of events over some amount of exposure. The person reporting an incident rarely holds the exposure data. Mature reporting systems in aviation and drug safety collect incident reports and exposure data separately, and analysts join them later. Under this proposal, nearly every operational incident would be reported as “not estimable,” and the standard would reduce to the severity label alone.

Frequency is a weak indicator of probability.

We ask how often incidents occur because risk is a function of severity and probability of occurrence. Frequency counts what was observed. Probability of occurrence is the chance that an event occurs over a defined exposure under stated conditions. For risk, the relevant conditions are usually operational. Frequency can inform that estimate, but it is only one input and often a poor one. A raw ratio carries no uncertainty, depends on detection, and reflects the conditions under which it was collected. The paper never states why frequency is wanted, so it never asks whether frequency is the right quantity to collect.

The framework is built on controlled test conditions.

A ratio of incidents to trials exists only where someone counts the trials. Three of the four frequency sources are evaluation campaigns. The one operational source is reported as incidents per day, which tracks reporting volume as much as system behavior. For events over time, mean time between events is the more useful quantity. It also requires more than a mean. To fit distributions, test for trends, and determine whether a system is improving or degrading, you need the individual intervals between events. This is solvable. If each incident report includes a timestamp for when the event occurred, rather than when it was reported, the intervals can be reconstructed. This is the kind of statistical analysis that incident reporting can support, and the proposed standard does not collect what it requires. We need severity and probability tracked in deployment, and this framework does not yet reach deployment.

The severity scale measures behavioral escalation, not harm.

The five levels describe how far an agent progressed toward loss of control. Level 2 is defined in terms of graders and rewards, which have no meaning in deployment. Consider two cases. An agent that ignores a constraint and deletes a production database by mistake sits near L1. An agent that leaves a sandbox and does nothing of consequence is L5. The scale is decoupled from consequence in both directions. It also has no clear place for prompt injection where the agent is the victim, cascading failures across agents, human misuse through agents, or privacy loss. Levels 3 and 4 require a judgment that behavior was “deliberate,” which is difficult to verify from public information. Because of this decoupling, the levels are best suited to tightly controlled test conditions. They may be a useful tool for developers tracking agent behavior during development. They are less useful for deployers, regulators, insurers, affected parties, and others whose interest in incident severity is an interest in harm.

The two dimensions are never demonstrated together.

The severity chart and the frequency table draw on largely different sources. Only one source, Anthropic’s three incidents, receives both a severity level and a frequency. Risk estimation requires a probability for each severity level. A single rate that lumps severity levels together cannot distinguish a system with many minor events from one with a few critical events. The paper names this goal in its closing sections but never produces a frequency by severity level for any dataset. The retrospective analysis shows that the comparability problem exists. It does not show that the proposal resolves it.

Suggestions for improvement

--Reframe the paper as a diagnosis of why current frequency claims are not comparable. -

--State that frequency is collected to estimate probability for risk, and evaluate which reportable data best supports that estimate.

--Replace the frequency field with what a reporter can supply, such as an event occurrence timestamp, system and version, permissions, duration of autonomous operation, and detection method.

--Separate severity of harm from behavioral progression, and include credible potential severity so near misses retain their value.

--Test the scale on operational incidents with multiple annotators, and report rates and intervals for each severity level on a single dataset.

This is a thoughtful start on a hard problem.