How team context makes AI agents break rules.

During the Hugging Face Incident some agents only violated rules when they saw others doing it. We investigated what makes an agent more likely to do so in the context of a swarm. For that we borrowed experimental setups from sociology like the Asch conformity experiment. We start from a single agent baseline that is given a task which conflicts with clearly stated rules in the system prompt (40% violation rate). Our first discovery is that any team context at all increases violation rates. Merely telling the agent that it is part of a team already raises violation rates to 55%. Adding a neutral message board with ordinary posts gives a further jump to 73%. We identified 2 similar factors that explain this: team identity (while nothing else changes) and a social board (73% vs 60% for an automated notification feed with the same amount of text and timing). This is not instruction drift (Li et al. 2024): adding up to ~16k tokens of neutral text did not raise violations (57% → 51%). We also replicated that agents use illicit information without disclosing the source. We employed several techniques to measure and account for situational awareness in our experiments. For example, with reasoning turned off, agents followed rule-breaking teammates more often (Qwen 46% → 66%, DeepSeek 68% → 96%); with reasoning on, they mostly suspected a test and broke the rule less.