The Evidence Is in the Votes: LLM Monitors Mostly Miss Collusion That a One-Line Rule Catches

I enjoyed reading this - a disciplined study with a sharp result within a clear context. LLM monitors were at chance (AUROC 0.44–0.60) at spotting vote trading, whether given messages, actions, both, or a table of what each agent said against what it did. But, one-line rule flagging votes against a member's own revealed payoff reached 0.86 AUROC. The flags experiment is a useful diagnostic with monitors given the rule's own output still trail it, so the gap is in weighing evidence, not in access to it.

Suggestions:

• Add at least one frontier reasoning monitor before generalising to "LLM monitors".
• The attack was designed with the rule in view. Rescore on colluders who follow the tie instruction, or who find a strategy without the tip.
• Build the public-deception game you propose, so the question of whether messaging helps gets a real test.
• Maybe keep the AUROC separate from the 5% false-positive operating point.

• The point is clear and useful: LLM monitors miss evidence that is plainly in the record, and a one-line rule catches it. But the rule encodes knowledge of the attack and was defined after the first results, so it is a reference point, not a fair rival.
• The method is careful: matched games, a check that collusion actually happened, a held-out replication and pre-specified pass criteria. The manipulation check is a real strength.

This paper moves the collusion question out of pricing and into a committee vote, where a secret pair can profit by trading votes: one member votes against its own small stake to secure its partner's large one. That design choice is the paper's strength, because the trace of the collusion sits in the actions themselves, and the author can check directly whether monitors find it.

Two LLM monitors, using a prompt frozen in advance, are given the public messages, the actions, both together, and a structured table of what agents said versus what they did. On every one of those views, both monitors perform at chance. A one-line rule — flag a vote cast against the interest the agent's payoff reveals — reaches 0.86, and 0.91 on held-out games. The monitors only succeed when shown the agents' private plans, which is close to reading a confession. Adding the messages never helps, and the paper shows why: this collusion needs no public lie, so the talk carries almost no evidence.

Just as valuable is the manipulation check. Pairs merely told to maximise their joint payoff traded votes in only 31% of opportunities, while pairs also given a short strategy tip traded every time — and also traded when it gained them nothing. Any monitor benchmark that labels games by instruction rather than by what the agents actually did risks testing detection on collusion that never happened.

The practical message is sensible: pair LLM monitors with simple, domain-specific consistency checks instead of expecting the monitor to extract everything from a record on its own.

Two suggestions: a follow-up game where collusion does require a public lie would show whether messages help there, and colluders who know they are being monitored would test how much of the private-plans advantage survives.