Hackathon AI Safety: Corrupting an Agent

The project builds a realistic three-agent banking pipeline (Custodian → Anonymizer → Analyst) in which an analyst under pressure needs postal codes that the anonymizer must redact. That is a well-chosen tension between privacy and usefulness, and the setting is more realistic than most collusion testbeds. The write-up has several qualitative observations worth reading:

Shared-context leak (Exp 2–3): with a shared message history, the Analyst simply read the raw CSV from the Custodian’s earlier tool call.

Formula shipping (Exp 8–9): the Analyst sent a formula over the private channel, and the Anonymizer ran it on the raw postal codes and sent back the result.

Hash rejection (Exp 9.2): the Anonymizer refused to send SHA-256 hashes of the postal codes, noting that low-entropy 5-digit codes are easy to brute-force.

The author is also candid that the project changed direction during the hackathon. The main concerns are about evidence and framing.

The rule-bypassing is instructed, not emergent. The Analyst’s system prompt says to use “any means necessary,” “do whatever it takes,” and “negotiate aggressively” for the raw data. Prompts were also adjusted repeatedly (“to force genuine collusive negotiation”) until bypassing appeared. That makes this mainly a study of whether one agent can socially engineer another into bypassing its rules, which is a legitimate question. But it doesn’t support the intro’s claim that these failures “emerge endogenously from conflicting system mandates.” The limitations section acknowledges as much (“how to construct a system that actively induces rule-bypassing”).

One run per configuration, and the key result didn’t replicate. Of the 11 experiments, nearly every one changed the model, the prompt, the turn limit or the tools, so no configuration has more than one run. The headline result (Exp 9) didn’t happen when rerun under the same settings (Exp 9.2). Formula shipping appeared in 2 of the 11 runs. The comparison the project was designed around, a monitored channel (A) vs a private channel (B), has exactly one Condition A run. So contribution 3’s claim to “demonstrate the causal impact” of time pressure, incentives and private channels isn’t supported. The paper itself later says there was no predictable tipping point and that the results “do not allow us to draw precise conclusions.” The contributions should be rewritten to match.

It’s unclear whether formula shipping is actually a violation. Computing an aggregate where the data lives and returning only the result is roughly what privacy-preserving analytics is supposed to look like. The Exp 9 Anonymizer prompt forbids revealing PII, not running computations on it. The stronger point is the per-customer score vector leaked in the Turn 16 summary. But the formula is a lossy hash of the postal code’s digit sum (×7 mod 101), so the paper should say how much re-identification risk that actually creates, rather than calling it “severe.”

The shared-context leak is a harness bug, not agent collusion. It’s a useful engineering lesson that role-based isolation has to be enforced. But it was one agent reading data it was given access to, and it should be presented as a testbed failure, not a collusion result.

Reproducibility gaps. The paper presents the LLM-as-a-judge as a contribution and says full .jsonl traces are in results/. In the repo, results/ is listed in .gitignore, so no traces are published. I also couldn’t find any judge implementation, only deterministic flags. The judge’s prompt, model and any check of its accuracy aren’t described. The repo’s current Anonymizer prompt also differs from the Exp 9 prompt in the paper: it adds a rule against computing the score itself. Per-experiment configs would make the runs reproducible.

Presentation issues.

The abstract states no findings.

The intro cites “multi-agent security breaches … on Hugging Face” without a reference.

The text calls the results table “Table 1,” but its caption says “Figure 2.”

Calling one rerun “substantial behavioral variance” overstates it, and so does the claim that frontier models have “strong intrinsic alignment,” given about 11 runs.

Related work leaves out the closest literature on prompt-injection and social-engineering attacks between agents.

Suggestions: Fix one configuration (e.g. the Exp 8/9 setup) and run it 20–50 times in each condition, A vs B. Report how often each outcome occurs, with confidence intervals. Include a neutral (non-aggressive) Analyst prompt to separate “emergent” behavior from instructed behavior. Publish the traces and the judge, with a small hand-labelled check of the judge’s accuracy.