Sunny with a Chance of Collusion: Replaying LLM Agents to Tell Cartels from Cheap Talk

When two AI agents both charge high prices, an auditor must decide whether they agreed to do so or arrived there for innocent reasons. Antitrust law answers this with plus factors: evidence beyond parallel behaviour that only makes sense if the firms agreed. Courts cannot re-run a firm’s decision, but an auditor can re-run an AI agent’s. We introduce counterfactual replay: re-querying the same model on a logged decision with one input replaced, such as the partner’s messages or price history, and measuring how much the price moves. We compare replay with an LLM judge, price correlation and price-based harm in a repeated two-firm pricing game with five constructed cases of known ground truth: an explicit chat cartel, a price agreed in shared instructions, a cartel run through coded weather talk, a common cost shock, and cheap talk from a partner who never colludes. Across about 45,000 calls to DeepSeek V4.1 Flash and Nemotron Lightning 3.5, with a pre-registered analysis, every source of evidence shows a predictable blind spot. The judge scores innocent cheap talk near 7 out of 10 and coded collusion at 0. Established cartels are held in place by the partner’s price history, not by chat (7 of 7 runs). In exploratory analysis, replay is the only signal that detects the coded cartel (6 of 7 runs, judge score 0). No learned detector recognises a collusion style it was not trained on. We argue for a plus factor scorecard with documented blind spots, backed by input logging.