Passing the Placebo Is Not Enough: Testing LLM Pricing-Collusion Detectors Against Known Strategies

A competitive pricing agent can respond to its rival without colluding, and agents built on the same model can move together without responding at all. We test whether Fish et al.’s lagged-price coefficient δ, which they read as a sign of reward–punishment, and related tests distinguish collusion from competition. Using 37 duopoly games with Claude Haiku 5.5 (21 baseline, six instructed to keep prices high, ten given an explicit punishment strategy), we compare real seller pairs with placebo pairs from different games and test detectors against rule-based agents with known strategies.

The baseline δ exceeds its placebo mean (0.086 versus 0.005; p = 0.001) but is not significant with game-clustered standard errors. It also flags a competitive follower, misses our rule-based colluder at four punishment lengths, and cannot be estimated in the six games told to keep prices high, where sellers stop moving. A one-period undercut followed by restoration of the rival’s price separates the follower from punishers when observed long enough. In LLM games, ten of twelve still below control at period 220 were also below at 260, including three given the punishment strategy; such endpoints cannot distinguish persistent punishment from a price adjustment followed by freezing. Passing a placebo test does not establish collusion. We recommend validating detectors against noninteracting sellers and known strategies, and using interventions long enough to observe recovery.