No Smoking Gun Without a Placebo: Four Conditions for AI Collusion Evidence

framework featuring four executable conditions and a randomized canary test that successfully demonstrates how LLM price tracking stems from prompt compliance rather than true collusive profit gain. However, its empirical evaluation remains constrained.

The four conditions and the executable checker are the contribution. They are a good one: PASS / FAIL / NOT‑ASSESSABLE from released files is something other teams can run on their own work and treating NOT‑ASSESSABLE as a finding is fair. The ColludeBench leave‑one‑out (15/17 to 7/17 without a statistic that fires on 99% of windows) and the n = 7 interval recovery for the CoT‑judge null are arithmetic anyone can verify. The Z2 compliance result is the empirical finding: the follower copies any number, 1.20 to 3.50 from any speaker so the lockstep is compliance rather than collusion. Retracting five claims after independent review is the right practice. The canary however cannot carry the weight the abstract puts on it: with MDE 0.30–0.56 price units against a positive‑control effect of 0.13 "no cost leak" is true but uninformative and the M‑S arm is post hoc and cannot separate exposure from framing. Condition (i) on NARCBench is, by design since the role is the treatment; the behavioural‑label reanalysis is the interesting part and should lead. Most audit cells are NA so "every checkable condition fails" is a claim than it reads. Checker thresholds (0.95, 0.20 10 units) need justification.

Strengths. The main contribution is turning "what counts as collusion evidence?" into four checkable conditions with an executable checker that returns PASS / FAIL / NOT-ASSESSABLE from released files. Treating NOT-ASSESSABLE as a finding (the instrument doesn't release what a reader needs) is a useful norm, and the checker is something other teams could run on their own work. The audits are concrete and verifiable: removing permutation_invariance drops ColludeBench's union from 15/17 to 7/17 while that statistic fires on 99% of honest windows (Figure 1 makes this clear); the Riemer et al. CoT-judge null rests on n = 7 with a CI of [-0.65, +0.83]; NARCBench's positive label equals the assigned role, and the behavioural reanalysis (8/31 positives did not coordinate) is the more interesting part. The canary is pre-registered and hashed, amendments are dated, failed runs are archived, every prediction is reported whichever way it fell, and five sprint claims were retracted after independent review. That is unusually good practice. The Z2 follow-up is the clearest empirical result: B copies a proposed number from 1.20 to 3.50, including below Nash, and follows a neutral analyst almost as closely as the rival (shift ratio 0.97), so lockstep pricing here is compliance rather than collusion. Separating price level, profit index and information flow as three different questions is a valuable framing.

Weaknesses. (1) The canary's nulls are underpowered: MDE 0.30-0.56 price units against a positive-control effect of 0.13, and prediction 4 fails in M+P. The canary shows the standard can be met, but "no detectable cost leak" says little, and the abstract should carry that caveat. (2) The thresholds (AUROC >= 0.95, TPR drop > 0.20, < 10 units, |r| >= 0.5) are not justified, and there is no sensitivity analysis. Condition (ii) could also penalise legitimately complementary ensembles; the real problem with permutation_invariance is its honest-window firing rate, which is a placebo/FPR issue. (3) Eight of the twelve audit cells are NA, so "every condition that can be checked fails" rests on four FAILs across three instruments. On NARCBench, label = role is partly by design when the role is the treatment; framing it as label vs. behaviour would be fairer and stronger. (4) Generality is limited: one model (Qwen3.8-27B), one temperature, one prompt per arm, 8 rounds. Prices above monopoly with lower profits suggest the agents are not optimising well, which limits what the testbed says about collusion. Z2 tests round 1 only, so it cannot speak to coordination sustained by punishment in later rounds. The M-S arm is post hoc and cannot separate exposure from framing. (5) Presentation is dense: arm codes (P-0, POS, M, M+P, M-S, Z2), many appendix pointers and a small Table 3 make it slow to read. The 4.3 worked case deserves more prominence.

Suggestions: a power analysis to set run counts, threshold sensitivity for the checker, a multi-round Z2, applying the checker to more instruments, and sharing the findings with the audited authors so NA cells can become PASS/FAIL.