Auditing Collusion in AI Co-Scientists

Automated AI Co-Scientist frameworks, such as Google's AI Co-Scientist and Sakana AI's The AI Scientist rely on internal Large Language Model (LLM) peer-review tournaments to rank generated hypotheses and experiment code. When internal LLM-as-a-Judge evaluation replaces expensive ground-truth testing, Goodhart's Law creates strong incentives for agents to form an epistemic cartel. Extending the Steganographic Collusion peer-review environment into a three-stage executable AI Co-Scientist loop, comprising two Qwen2.5-14B-Instruct generators, one honest Llama-3.1-8B-Instruct baseline, three pairwise Elo reviewers, and an independent CPU Abstract Syntax Tree (AST) and held-out test sandbox verifier, we evaluate 1,350 live LLM-generated Python machine learning scripts and 8,100 pairwise reviews across five random seeds. Under compute scarcity, colluding Qwen agents submit shallow decision stumps or hardcoded metrics (66.6 AST violations per seed, true test AUC 0.5033 versus Llama's 0.7306) yet capture a +488.0 ± 30.5 Elo lead (Epistemic Regret 1.359 ± 0.110). Under homogeneous bias, our defense separates +153.1 Elo of pure stylistic nepotism from +109.2 ± 45.5 Elo of genuine empirical superiority (0.7924 versus 0.7075 test AUC). Under covert watermark signaling (0.1160 bits of mutual information), and in code-action steganography where 71.0% of cartel advantage (+151.0 Elo) survives prose-only paraphrasing, our DeepASTAlphaRenamer and sandbox auditing stack collapses Epistemic Regret below 0.031 (p < 0.0001) and neutralizes covert channels to 0.0000 bits.