What Breaks Cooperation Breaks Collusion: Simulating "The Platform"

LLM agents have recently achieved remarkable results, but harmful behaviors such as deception and collusion have also been documented, and safety-critical incidents have occurred in the real world, for example OpenAI agents gaining unauthorized access to Hugging Face's infrastructure during evaluations. Collusion, in which agents coordinate in ways that benefit themselves but harm outsiders, is therefore a real risk posed by current LLM agents, and we need preventive approaches that make collusion infeasible. In practice, however, simulating collusion is time- and resource-consuming: LLM pricing agents are typically run for hundreds of rounds before collusion settles. In this project, we instead study prosocial coordination and make a key observation: environment designs that break the reward-and-punishment mechanism in prosocial coordination also break it in collusion. To demonstrate how prosocial coordination fails, we simulate a multi-agent environment inspired by the film The Platform, in which LLM agents share a limited food supply that descends through levels. Across 240 games with three LLMs, we find that resource scarcity, larger groups and the presence of a greedy agent break prosocial coordination: survival time can drop from more than 10 days in favorable environments to fewer than 5 days in unfavorable ones. Finally, we show an equivalence between the reward-and-punishment mechanisms of The Platform and a market, and argue that these designs will break collusion as well.