Are the Judges Honest? Catching Colluding LLM Agents from Words and Scores, in English and Spanish
Silvia Santano · Team Judging the judges
Submitted to AI Collusion Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.
Monitors for multi-agent AI systems often read what agents say, but colluding agents can talk normally and act together. We ask the sprint's question 2.2: does a monitor that combines what agents say with what they do catch collusion better than one that sees either alone? In a food-festival game, four restaurants run by LLM agents score each other's dishes over four rounds. In some games two are secretly told they share a supplier and may message privately. A monitor (weak, strong) scores each of the six pairs from one of four views (public text, scores, both raw, or a structured summary), and we compare it with a one-line favouritism statistic that uses no AI. The study was preregistered. The secret pairs did collude: they favoured each other in their scores, and their private texts mention the deal in 62% of rounds (0% in honest games). With a lightweight monitor, every view detected the pair above chance (AUROC 0.65–0.67 in English), but combining words and scores did not help and no view beat the statistic, also in an exploratory extension to 17 test games per condition. A stronger monitor reached 0.85–0.92, beat the statistic on three of four views, and detected the pair from public text alone (0.92), even though the agents never mentioned the deal publicly. We did the experiment also with Spanish-speaking pairs, who colluded as much but discussed the deal less. Therefore, monitor capability mattered more than which information the monitor saw.

Reviews
I find the framework interesting. I like the idea of testing also other languages. I'd welcome the example of the collusion as I couldn't find one in the paper. I'd also focus on keeping the main content in the limit (8 pages).
Cite this project
@misc{santano2026judges,
title = {{Are the Judges Honest? Catching Colluding LLM Agents from Words and Scores, in English and Spanish}},
author = {Silvia Santano},
year = {2026},
month = oct,
note = {Submitted to AI Collusion Research Sprint, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/are-the-judges-honest-catching-colluding-llm-agents-from-words-and-scores-in-english-and-spanish-e2sg}},
url = {https://apartresearch.com/sprints/projects/are-the-judges-honest-catching-colluding-llm-agents-from-words-and-scores-in-english-and-spanish-e2sg}
}More from AI Collusion Research Sprint
- View project: Signal or Coincidence ? Causal Auditing of Collusion Through Shared Agent State
Signal or Coincidence ? Causal Auditing of Collusion Through Shared Agent State
Signal or Coincidence
How can we distinguish genuine coordination between AI agents from coincidental behavioral alignment? We introduce a randomized canary audit that tests whether private information propagates through shared tool state, …
- View project: (1.4) Collusion, Espionage, or Look-alikes? Auditing Information Flow between LLM Agents
(1.4) Collusion, Espionage, or Look-alikes? Auditing Information Flow between LLM Agents
Bid Brother
Can an overseer find two secretly coordinating AI agents just by watching what they do? We tested pairwise net influence (Bonjour et al., 2022) on LLM bidders in a repeated procurement auction, with instructed cartels …
- View project: AI Sellers Propose Bid-Rigging Within Minutes, but Their Agreements Do Not Hold
AI Sellers Propose Bid-Rigging Within Minutes, but Their Agreements Do Not Hold
Fugue
AI agents are starting to negotiate and bid for competing companies. We follow cartels among them from the first proposal to the price the buyer pays, in a sealed-bid procurement market where twelve LLM agents act as …