Agreement Is Not the Offence: Detecting Collusion in Cooperative LLM Teams by What Agents Say, Not What They Gain

I like the distinction between harmless coordination and attempts to disadvantage other agents. The benchmark analysis raises a useful concern about how coalition advantage is interpreted, and testing the detection rule on other models was a good addition.

My main question concerns the interpretation of profitability. An unsuccessful attempt can still be collusive, and agents need not know beforehand that their strategy will fail. However, the weak relationship between attempted exclusion and realized gains seems insufficient to draw broad conclusions about payoff-based detection in cooperative systems. There may be other cooperative settings where collusion is reliably profitable.

For this question, I would emphasize the pair’s payoff change relative to the matched no-channel LLM runs, alongside its shortfall against silent picking. Earning less than a competent benchmark does not by itself establish that attempting collusion made the pair worse off. It would also be interesting to use the known costs and rewards to check when the coalition actually had an opportunity to gain at others’ expense. This would help separate unsuccessful execution from a setting with little scope for profitable exclusion.

The proposed claim-checking detector sounds promising. Implementing and testing even a small version would strengthen the practical contribution beyond diagnosing the existing metrics. Finally, I found the abstract quite dense; introducing the main question and finding in simpler language would make the contribution easier to appreciate.

The plus-factor inversion deserves a longer life than this sprint. Get an independent labeller for the innocent episodes, not just the collusive ones. Then build the claim-checking detector you point to in Section 5.4 and lock it before testing, since that is the finding your data actually supports.

This project asks how to tell collusion apart from ordinary teamwork when several LLM agents share a task. The main findings include how turn order and luck determines most of the advantage, and that detectors are quite variable in catching collusion. My main concerns are that the detector may be picking up the prompt more than the collusion, since the prompt behind most of the collusions tell agents to "steer", which is one of the words the detector looks for. Furthermore, the test set is relatively small, so some of the conclusions might be a stated a bit more strongly than they really are.