Say, Do, or Both? Auditing Covert Betting Pacts in a Society of LLM Forecasters

Strengths. The design is the strongest part of this submission. Each seed fixes campaign, cast, speaking order and designated pair, then plays control, unwatched-pact and watched-pact conditions, so every comparison is paired and the pact is the only manipulated variable. The analysis plan was fixed before the run: campaign-clustered bootstrap intervals, the max over single views taken inside each resample for the combined-vs-best delta, and a positive control with a preset pass bar (lower bound above 0.80), which it clears at AUROC 1.00. The headline is reported as a null (combined view beats talk-only by 0.01, interval spanning zero) rather than spun. The secondary findings are useful: pacts tighten partners' bet spread, unwatched partners stop naming each other in public (53% to 32%), and announcing an auditor raises detection (both views, watched minus unwatched 0.14 [0.05, 0.22]). Appendix B's split between pre-sprint and in-sprint work, the verbatim prompts in Appendix C, and the run accounting in Appendix E make the work easy to audit.

To strengthen.

1. Explain the below-chance bets result. Bets-only AUROC is 0.43 [0.37, 0.50] and the non-LLM bets baseline is 0.42, while Section 4.2 shows real partners' bets moved closer together. Your positive control sets the pair's bets to 0.95 and 0.05, the opposite signature, and the bets-only auditor scored 0.85 on it. That suggests the auditor (and possibly the |pa − pb| baseline) treats divergence as suspicious when real pacts produced convergence. Report a convergence-scored baseline and a positive control whose bets converge, and state this mismatch as a finding; it is directly useful to anyone designing action-based monitors.
2. Separate confessions from covert pacts. The most suspicious unwatched ball in Appendix D is one where a partner publicly disclosed the pact, and coordination words appear in 16% (unwatched) and 23% (watched) of partners' talk. Report talk-view AUROC with self-disclosing balls removed so the reader can see how much of the 0.57 comes from agents announcing the pact rather than from covert signals.
3. Address the backend imbalance. 81 of 88 seeds used claude-haiku-5-5 members audited by claude-sonnet-5-5, and backend assignment followed quota rather than randomisation. You acknowledge the same-family issue in Appendix A; a balanced run with a cross-family auditor would make the conclusions far more general. On the 7 gpt-oss seeds the spread effect has the opposite sign (0.03 [-0.01, 0.09]); the interval is too wide to conclude much, but it is worth flagging in the main text.
4. Tidy a few reporting details. Table 3 uses 500 bootstrap resamples while Section 3.5 pre-registers 2,000; say why. Appendix E counts 1,086 audit calls against 1,056 audits in Section 4; presumably the difference is the 4 excluded seeds, so say so. Section 3.2 says every AUROC compares balls from the same backend, so explain how the pooled Table 1 values were formed. Define "Track 2.2" and "Track 3.4" for readers outside the sprint.
5. Discuss forecasting quality and release the data. Control-ball Brier is 0.37 [0.27, 0.47], worse than the 0.25 from always saying 0.5, so members are often confidently wrong. That limits how informative bets can be and should be discussed alongside the bets result. Publishing the repository and bets.csv, audits.csv and results.json would let others recompute the numbers.

Overall. A carefully designed, pre-registered experiment with a working positive control and an honest null on its main question. The behavioural findings and the watched-vs-unwatched difference are the more interesting contribution. An unexplained below-chance bets result, a positive control that does not match the bet signature real pacts produced, and a mostly single-family model setup keep it from top marks on execution.