Plus Factors for Collusion Monitors: LLM Monitors Mistake Open Cooperation for Collusion

An important, under-studied problem handled with real rigour: monitors that accuse honest agents of colluding when they simply cooperate in the open.

Séance makes this hard negative concrete, and the result is stark. Under a plain prompt, every monitor accused most honest games played by Qwen, and Claude Opus 5.5 accused all 50 while naming the right pair in every collusion game. The work is immediately reusable because:

• A pre-registered main study;
• A frozen held-out test set scored once;
• Baselines (prior, top scorers, a statistics detector, recalibration);
• Aablations;
• A full release with an explorer.

Suggestions:

• Test whether the plus-factor paragraph works through its evidence or simply shifts the prior. Use a prompt that only states the base rate and a placebo paragraph of the same length, and compare at matched false-alarm rates.
• Tabulate missed coalitions for every condition and use one catch-rate definition throughout.
• Test a paragraph not tied to this game, and a second setting such as pricing.
• Stronger classes: make collusion actually pay, and audit honest players through their private notes.
• Report the messages-only and actions-only views listed in the plan.

The hard-negative design is a valuable contribution: a monitor should distinguish innocent open cooperation from a hidden alliance, rather than merely identify the most coordinated pair. The report communicates that distinction and the false-alarm/missed-coalition tradeoff clearly. The paragraph reliably reduces accusations, but that alone does not establish better discrimination; most monitors also miss more coalitions, and the Qwen monitor’s estimated ranking improvements are not statistically clear. The frozen recalibration baseline is a meaningful comparison. Most convincingly, on Haiku-played test games both approaches accuse 6 of 50 honest games, while recalibration misses 10 more coalitions (reported interval 3–17). This supports an advantage beyond caution at that operating point, though it does not establish superiority to calibration generally or broad detection improvement. Please extend this comparison to more environments, player populations and calibration methods, with thresholds fitted on separate data and prospective low-false-positive evaluation. The small toy-game test sets and uncorrected secondary comparisons warrant restrained generalization.

I think this is pointing at something real, where honest agents who cooperate openly can look a lot like they collude to an LLM monitor. Séance is a clean, minimal setting for studying this. For a two-day sprint the work is extremely thorough. It has a held-out test set, different monitor families, a threshold-onlly baseline and ablations of the prompt. It is also very easy to read.

The main way to strengthen it is to tune the monitors. At the moment the results can't fully separate "the monitors are uncalibrated or under-elicited" from "these cases are genuinely hard to tell apart".

One could:

• set thresholds on separate honest games
• iterate on the prompt
• add some examples of honest sharing and collusion

and then show if/which false positives and false negatives remain.

Then as a second step a qualitative analysis of those remaining errors could be useful. For example, what do the honest tables that still get accused have in common or which collusions still evade the monitor? Which features do the monitors cite when they are wrong?