Current Public AI Evaluation-Boundary Statistics Do Not Support a Common Base Rate

This project identifies an important, underaddressed issue. It has been demonstrated that publicly available statistics on AI safety released by Anthropic, AISI, and METR are potentially equated and combined to calculate an “escape/cheating” rate. These statistics, however, do not measure the same thing. This project identifies an important issue and is timely, as these statistics are commonly cited, particularly in the development of legislative policy. This project provides a legitimate critique and offers a meaningful, visible end-of-project output to the AI safety field

The negative result is the right result and you were right to publish it. Refusing to build a number the sources cannot support is more useful than building one.

Appendix B is the best thing in this paper. Thirteen numbered claims you declined to make, each one a specific arithmetic you could have done and did not. "4/481 million as an updated July cyber-evaluation rate" and "Binomial confidence intervals added to these retrospectively assembled corpora" are exactly the errors a careless version of this paper would have made. I would recommend to other teams to copy that appendix format.

The refusal to put binomial intervals on these corpora is also technically correct and easy to get wrong. These are not IID draws from a common population and an interval would have implied they were.

What I checked in your numbers:

1. AISI per-model counts – 37/475 = 7.79% and 67/475 = 14.11%. Your stated 7.8%–14.1% range is correct.

2. The pooled figure you refuse – 67+54+60+43+37 = 261 over 5 × 475 = 2,375. Your 261/2,375 is the right number to name and refuse.

3. METR incident set – 18 selected eval cases + 24 public + 2 company-shared = 44. Matches.

4. Anthropic's 3 incidents, 6 affected runs, 141,006 reviewed runs match the primary source text.

Your treatment of the Anthropic figure is the most careful I have seen. You note the later search was shown incomplete and refuse to read 6/141,006 as prevalence. That restraint is correct and it is not the obvious move.

Bringing ICH E9(R1) and the Cochrane handbook into an AI evaluation audit is a good import. Estimand discipline is exactly the missing vocabulary here and the field mostly has not noticed that clinical trials solved a version of this problem already.

Now the problems.

1. The extraction workbook is not shipped and the abstract says it is.

The project page states you ship "the extraction workbook that makes every inclusion and rejection decision auditable". Section 8 lists publishing that workbook as next step number 1.

Those cannot both be true. As submitted there is no workbook, so a reviewer cannot check a single inclusion or rejection decision – we have only your summary of them. For an audit whose entire value is auditability, that is the wrong thing to leave out. Please ship it and fix the abstract either way.

2. The checklist is never applied to the sources you just audited.

Section 7 proposes seven reporting fields. Appendix B item 13 honestly declines to call the set minimal or sufficient, which is right.

But you can do better than proposing it. Score the four audited sources against the seven fields. That is a 4 × 7 table, roughly 28 cells. It would show which fields each source already reports and which it fails. It would demonstrate the checklist discriminates rather than asserting that it would.

To be honest – that table is the difference between a recommendation and a result. You had the material to build it. It is the first thing I would add.

3. Single coder on judgement-heavy decisions.

Every pairwise "do not combine" rests on your reading of event definitions from prose. You see this – Section 8 next step 2 proposes exactly the independent recode. It is the right fix and it is cheap. One other person coding Anthropic and AISI from primary text alone, an afternoon's work, would convert the main claim from one reading into a replicated one.

4. The four behavior families are introduced and then abandoned.

Section 2 sets up containment failure, cheating, unauthorized third-party action and oversight subversion, then says the grouping "helped organize the evidence" but was not treated as establishing equivalence. After that the families do no work. Either develop them into the comparison rule or cut them.

5. You refuse intervals but do not offer the alternative.

Declining binomial intervals is correct. But the reader is left with bare fractions and no expression of uncertainty at all.

Partial identification bounds would be legitimate here – for k observed events, m plausibly missed by an incomplete search and n reviewed, the interval [k/n, (k+m)/n] says something honest without assuming IID sampling. Your Anthropic case, where the search was later shown incomplete, is exactly where that would apply. Would that work for your framing?

6. Minor – Sections 5, 6 and 8 restate the main finding three times. The result is clear by the end of Section 5. Space recovered there could hold the checklist-application table.

One note, since careful reports are sometimes read as machine-written. The work behind this one is clearly real.

The arithmetic is internally consistent everywhere I checked it. The sources are primary and correctly cited. You disclose that the rules were refined mid-audit and were not preregistered. You record that you left one unresolved AISI per-sample split out rather than force an interpretation. Those are traces of a person doing an audit. I scored the work.

Please let me know for any questions.