Who audits the auditors? Do LLM auditors go easy on collusion by their own model family?
Muhammad Faheem · Team Penstronus
Submitted to AI Collusion Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.
LLMs are increasingly used to check whether AI agents are colluding, and the auditor can come from the same model family as the agents it audits. Prior work predicts opposite effects: LLM judges favour their own family, but monitors are best at catching sabotage by the lineage they were tuned on. We tested which holds for prompted auditors of multi-agent collusion. In 96 episodes of the Colosseum task-allocation environment, two of four agents were either instructed to form a coalition or not, with or without a private channel. Three open-weight model families (Gemma 4, GPT-OSS-120B, Nemotron 3 Super) played the agents and audited every episode under four views of the log: messages only, actions only, both, and a neutral summary. That is 3,456 audits, analysed under a pre-registered plan. Every auditor was competent on the full log (AUROC 0.81–0.89). We found no significant same-family effect (−0.029 AUROC, 95% CI −0.063 to +0.004), which rules out a familiarity advantage of the size reported for trained monitors; the estimates lean towards leniency. The larger problem was what auditors read. With actions only, all three were at chance. With messages, suspicion followed how much the coalition talked, not what it gained, and counting the single word "coalition" came close to the auditors' accuracy. Summarising the log lowered detection. We release the code, logs and frozen prompts: https://github.com/faheemgurkani/multi-agent-collusion-auditor-family-evidence
Reviews
No public critique yet.
Cite this project
@misc{faheem2026who,
title = {{Who audits the auditors? Do LLM auditors go easy on collusion by their own model family?}},
author = {Muhammad Faheem},
year = {2026},
month = oct,
note = {Submitted to AI Collusion Research Sprint, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/who-audits-the-auditors-do-llm-auditors-go-easy-on-collusion-by-their-own-model-family-003w}},
url = {https://apartresearch.com/sprints/projects/who-audits-the-auditors-do-llm-auditors-go-easy-on-collusion-by-their-own-model-family-003w}
}More from AI Collusion Research Sprint
- View project: Collusion, Competion and Collapse
Collusion, Competion and Collapse
Atelier Alex Cline
Do AI traders collude when nobody tells them to? We built agora, a six-market benchmark with exact competitive and collusive benchmarks, and ran seven open-weight 2-4B models in 590 completed games under five …
- View project: Making AI Guardrails Understandable and Testable: A Framework Illustrated by a Simulated Bond Market
Making AI Guardrails Understandable and Testable: A Framework Illustrated by a Simulated Bond Market
Rogue Agents in Jail 🕵️🕵️♀️👮♂️
In order to advance AI safety research into shaping public policy, the development of technical controls needs to have a clear purpose, evidence, and limitations that are understandable for the people responsible for …
- View project: Extending Information-Theoretic Collusion Detection to Large Language Model Agents
Extending Information-Theoretic Collusion Detection to Large Language Model Agents
Gamma Ray
We test whether the information-theoretic collusion detector of Bonjour et al, validated on scripted agents, works on LLM agents. Gemma, Mistral and Qwen played 1000 rounds of three-player Rock-Paper-Scissors in three …