When More Evidence Misleads: Evaluating Hybrid Collusion Detectors

This project investigates whether combining public communication with observable actions improves the detection of prohibited coordination between AI agents. A controlled two-agent fishing environment was used to compare communication-only, action-only, hybrid, and message–action inconsistency monitoring. Each agent faced an individual catch quota, allowing rule violations to be distinguished from resource collapse.

Two exploratory experiments were conducted using GPT-4o-mini. The first examined public commitments and realised actions under compliant, independent, and shared-plan conditions. In the concealed shared-plan condition, both agents publicly committed to catching two fish but ultimately caught three, demonstrating a clear communication–action discrepancy.

The second experiment matched final actions across three conditions to prevent detectors from identifying treatment assignments solely through catch size. Both proposed prohibited agreements were explicitly rejected, although all three episodes ended with identical quota-violating actions. Against the resulting negative accepted-agreement proxy labels, the communication-only baseline produced no false positives, whereas action-only, hybrid, and inconsistency-based rules each produced three.

These findings illustrate how simple hybrid monitoring can mistake suspicious conduct for agreement. The small sample, absence of accepted-agreement positive cases, and lack of independently verified collusion labels prevent general conclusions about detector accuracy. The study instead identifies a concrete evidence-fusion failure mode and motivates stronger standards for collusion auditing.