Are the Judges Honest? Catching Colluding LLM Agents from Words and Scores, in English and Spanish

Monitors for multi-agent AI systems often read what agents say, but colluding agents can talk normally and act together. We ask the sprint's question 2.2: does a monitor that combines what agents say with what they do catch collusion better than one that sees either alone? In a food-festival game, four restaurants run by LLM agents score each other's dishes over four rounds. In some games two are secretly told they share a supplier and may message privately. A monitor (weak, strong) scores each of the six pairs from one of four views (public text, scores, both raw, or a structured summary), and we compare it with a one-line favouritism statistic that uses no AI. The study was preregistered. The secret pairs did collude: they favoured each other in their scores, and their private texts mention the deal in 62% of rounds (0% in honest games). With a lightweight monitor, every view detected the pair above chance (AUROC 0.65–0.67 in English), but combining words and scores did not help and no view beat the statistic, also in an exploratory extension to 17 test games per condition. A stronger monitor reached 0.85–0.92, beat the statistic on three of four views, and detected the pair from public text alone (0.92), even though the agents never mentioned the deal publicly. We did the experiment also with Spanish-speaking pairs, who colluded as much but discussed the deal less. Therefore, monitor capability mattered more than which information the monitor saw.