Talk Is Not Collusion: Do published collusion detectors catch LLM agents that collude on their own?
Rahul Kumar · Team Shinobi
Submitted to AI Collusion Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.
Collusion detectors for AI agents are usually tested on agents that researchers programmed to hide a secret signal. We tested whether a published detector suite, Audit the Whisper, catches language-model agents that collude on their own. In a simulated market where sellers set prices and can message each other, we ran 10 matched pairs of sessions. Groups of five GPT-4.1 sellers kept prices above the competitive level in all 10 sessions, while groups of three GPT-4.1 and two Claude Haiku sellers priced near cost in all 10 (exact paired test, p = 0.002). We scored detectors only on the GPT-4.1 sellers, so writing style could not separate the markets. Neither the suite’s detectors nor an AI judge using the market’s own monitoring prompt could reliably tell them apart. The judge did best, with a ROC AUC of 0.77 versus 1.00 for prices, because sellers in both markets used the same collusive language. A price rule using only the first five rounds flagged every colluding market with no false alarms. We also found that the suite’s three comparison methods, reported with perfect scores, are given the colluders’ identities as input. Monitors should therefore check outcomes as well as messages. All logs, code, and results are public, and one command recomputes the headline numbers.
Reviews
No public critique yet.
Cite this project
@misc{kumar2026talk,
title = {{Talk Is Not Collusion: Do published collusion detectors catch LLM agents that collude on their own?}},
author = {Rahul Kumar},
year = {2026},
month = oct,
note = {Submitted to AI Collusion Research Sprint, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/talk-is-not-collusion-do-published-collusion-detectors-catch-llm-agents-that-collude-on-their-own-0ax8}},
url = {https://apartresearch.com/sprints/projects/talk-is-not-collusion-do-published-collusion-detectors-catch-llm-agents-that-collude-on-their-own-0ax8}
}More from AI Collusion Research Sprint
- 1st placeView project: Monoculture Looks Like Collusion: A Replay Test for Hidden Information Flow Between LLM Agents
Monoculture Looks Like Collusion: A Replay Test for Hidden Information Flow Between LLM Agents
Collusion audits ask whether the actions of different agents depend on each other. Agents built on one model act alike without communicating, so these audits accuse honest fleets. Among four honest Claude Haiku agents …
- 2nd placeView project: Looking for a Hidden Agenda: Can a Model’s Activations Reveal Steganography that its Text Hides?
Looking for a Hidden Agenda: Can a Model’s Activations Reveal Steganography that its Text Hides?
Stego-probe
AI agents can collude without saying anything suspicious, just by choosing which true facts to mention. We built a hiring task where Qwen3-8B invents its own covert signals that get past an LLM overseer, then asked …
- 3rd placeView project: Do collusion probes generalise? Prompt confounds in NARCBench
Do collusion probes generalise? Prompt confounds in NARCBench
I investigated HELIOS, an activation-probe for detecting collusion. I found some apparent flaws, and analysed the (acknowledged) unreliable generalisation of the approach.