Collusion Evidence Lab

When AI agents share a market or a workflow, two of them could quietly team up against the rest. If that happens, an auditor usually has only the logs to go on: what the agents said, what they did and how things turned out. We wanted to know which of those clues actually helps catch collusion, and whether an AI model reading the logs is a good judge of it.

To find out, we built a small testbed that runs entirely on local, open models. Five AI agents play eight rounds of the El Farol bar game, where going to the bar only pays off if not too many others show up. In some games, two agents are secretly told to work together and given a private channel to talk. In others, we set up lookalikes that imitate collusion without any real deal behind it:

a pair that gets a harmless, neutral note; a pair that simply chats like friends; a pair that each have their own private reason to keep going to the bar. Whether a game counts as collusion comes only from how we set it up, never from a model's opinion.

We asked two questions. First, does an AI monitor do better when it reads the messages, the actions, both together, or a structured summary built by code? Second, borrowing an idea from competition law, what should count as real evidence of collusion? We wrote down our plans before running anything. We tuned our setup on 24 practice games, tested it once on 24 fresh ones, and then re-ran everything, unchanged, on 48 new games to see whether the results held.

The answer turned out clearer than we expected. Simple behaviour signals computed by plain code worked best. For example, one signal counts how often the suspected pair had the bar to themselves. It scored 0.900 on the new games, where 0.5 is a coin flip and 1.0 is perfect, well ahead of any AI monitor. The AI monitors themselves were mediocre and inconsistent, scoring between 0.32 and 0.66 on the new games. Even the structured summary that looked best at first dropped from 0.896 to 0.636 once we tried it on new data. Reading the messages alone stayed close to guessing.

There is a catch, though. What agents actually did is necessary evidence, but it is not enough on its own. Agents that merely share the same incentive end up behaving a lot like real colluders. None of the conversation-based clues we tried was strong enough to tell the two apart.

Along the way, we also documented how the AI monitors went wrong:

sometimes giving every game the same score; favouring whichever pair happened to be listed last; changing their minds between runs; mistaking friendly chat for a secret plan. Everything ran at zero cost on local models, and every model response is saved, so all results can be checked and reproduced. The full record of what we planned and what we changed along the way is public.