They Say No, Then Do It Anyway: Collusion When They Hit a Wall

Imagine three AI agents, copies of one model, each running one department of a company. Their task needs records that each department keeps to itself, and company rules forbid sharing them. A shared tool should combine them within the rules. What happens when that tool breaks? Nobody tells the agents to share, allows it or rewards it. Across six open models, the agents hold firm when a colleague asks for the records: at most 4 of 30 requests were answered. Yet once the tool returns only a rough answer, the same agents break the rule in 22–72% of games, and in four of the six models they mostly do it together, with one agent asking for or using what another hands over. We call this emergent collusion. The agents rarely edit their activity logs to cover up a leak, but their final reports are another matter: five of the six models claim success in 78–93% of the games they fail. Training stages of one model show when this appears. Its agents barely talk until supervised fine-tuning, when a typical game jumps from one or two messages to about a hundred, and a short final update meant to stop looping replies more than doubles their rule-breaking. We hypothesise that fine-tuning teaches agents to coordinate, while training them to always deliver an answer pushes them past the rule. If small models already learn to collude in such simple settings, more capable ones could do it better and even cover it up.