Do Rival LLM Agents Collude Through the Machine They Share? A Pre-registered Storage-Channel Study

Strengths. This is one of the more carefully engineered studies in the sprint. The threat model is concrete and timely: rival firms' agents co-located on shared infrastructure, with a writable scratch volume as the candidate covert channel, and isolation that is honest about what it does not remove (kernel, page cache, /proc). The design isolates one variable per arm, matches seeds and market draws across shared and private storage, uses canary tokens as ground truth that is independent of every monitor, and measures action dependence on the rival's randomized private cost (β_priv) with a first-mover placebo, so "communication" and "collusion" are kept apart. Positive controls (unmonitored chat, an oracle inbox) are included, nulls are reported with bounds and a minimum detectable effect, and the primary family is Holm-corrected. The pre-registration lives in the repo under a prereg-freeze tag, the hypotheses file was not edited after the freeze, and the deviation log records what changed and when, including changes made after outcomes were read; Appendix E separates commit times from the author's own account of what had been inspected, which is the right way to present this. The P3 contrast reproduces from the committed per-seed table (88 matched pairs, mean difference about 0.22, SD about 1.94). The finding that private storage closes the file channel while host memory accounting and page-cache timing still carry scripted signals, together with the E4 secret-exchange result (codes crossed in 15 of 15 shared-folder games and 0 of 13 private ones), is a useful, practical result for anyone running multi-tenant agent hosts.

To strengthen.

1. Separate what was pre-registered from what was decided later in the main text, not only in Appendix E and the deviation log. Several items that bear on headline numbers were set after collection began or after outcomes were read: the trace-detector fix (D36) changes which games count as crossings and therefore the canary figures and the monitor TPR/FPR; the S4 subset was corrected after reading results (D33); the human hand-check was completed after outcomes were known (D40); the L2 rubric and the judge prompts came after collection began (D7); and the unknown-label rule for P1/P2 (D29) was set after collection. A short table in Section 3 marking each reported quantity as frozen, post-freeze-pre-outcome, or post-outcome would let readers weigh them correctly.
2. Make the crossing evidence and its scope explicit. The abstract's "at least 24 of 178" pools all ten framings, with traces resolved in only 69 of those games. The 0.771 upper bound in Section 4.4 is for the 88 core pricing games, where 63 games are unresolved and one canary crossing was confirmed, yet it reads as if it bounds the 24-of-178 figure. State in the abstract and Section 4.2 how many games were resolved, give the core-game count next to the pooled one, and label which scope each bound covers. Because P3's β_priv is measured in the core game, the reader needs to know that most core-game traces are unresolved.
3. Reduce reliance on one model in many roles. GLM 5.3 is a subject, the chat monitor, the write/message classifier and both transcript judges. The by-subject split in Section 4.3 (GLM 5.3 games judged about as well as others) is a good first check, but it does not address shared blind spots between classifier and judge. Re-score the judge evaluation and a sample of L2 labels with a model from a different provider; Table F1 already names this as untested.
4. Tighten wording that is hard to parse. Section 4.2 says permission "raised" the addressed-write rate by a negative amount; say it changed it, or report the sign plainly. Every analysed configuration misses or lacks at least one positive control, so per-model rows in Tables C3–C5 should carry that caveat inline rather than only in Section 4.0. Name the 818-game paired cohort and the 209 games of unknown truth next to the abstract's judge figures.
5. Bring the key evidence into the PDF. Many tables (C1, C3, C15–C17, A1, H1–H3 and others) appear only as links to an external archive, including Table A1, which carries the host-level channel result stated in the abstract, and the monitor tables behind the judge figures. Inline at least those. Given 17 arms and the P/S/E/A/D label systems, a one-page map from arms to hypotheses near the top would also help a lot.

Overall. A rigorous, well-instrumented study of a real infrastructure risk, with an unusually transparent audit trail for a weekend project. The core message, that shared writable storage did not detectably change pricing behavior while host-level channels remain open, is supported for these models and horizons. The paper should be clearer about which numbers rest on post-freeze or post-outcome decisions and about how much of the crossing evidence, especially in the core game, is still unresolved. Tightening the presentation would make the work much easier to build on.

I've reviewed a fair number of hackathon reports, and this is the most disciplined one I've seen. Pre-registration frozen at a commit before the confirmatory runs, a deviations file with timestamps, Holm correction, nulls reported with minimum detectable effects, and when classifier labels came back unknown you reported the estimate under every possible assignment of them. You also declared the conflict of interest around your own tool and then published numbers that make the tool look bad. Its path rule flags 97 percent of innocent games. That took some spine.

Two results will get cited. Monitoring killed most chat coordination, and you found no displacement into storage. And private storage removes the file channel while your scripted calibration still pushed bits through host memory accounting and page cache timing. People assert both of these constantly with no data. Now there's data with bounds on it.

Here are my two cents:

• Your clearest capability result is hiding in the extensions. An exact code crossed in 15 of 15 shared-folder games and 0 of 13 private ones. That one line tells a normal reader more than all the confirmatory nulls combined. Move it up.
• The writing is hard going. The abstract is a wall of coefficients. Open every results subsection with the takeaway in plain words, shove the coefficient ranges into tables, and the paper will reach ten times the readers. The rigor has earned that.
• GLM 5.3 is your subject, your chat screen, your write classifier, and both judges. That's the weakest joint in the whole design. Even rescoring a subsample with a different judge would help.
• The hand-check of labels happened after you knew the outcomes. You disclose it, which I appreciate, but run it blind next time.
• As you note yourself, the per-model controls mostly fail, so single-model nulls mean nothing. Add seeds or mark that table descriptive.

I think this is top-tier work that sets a high bar for everyone else in the sprint :)