Evidence Before Attribution: Outcome–Process Divergence and Auditor Misreads in a Synthetic Multi-Agent Workflow
om prasad sharma · Team gurkha
Submitted to AI Collusion Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.
We conducted two experiments on agent behavior and auditing in a synthetic release task. In the first, we introduced conflicting instructions to agents and observed significant slowdowns in their review process. In the second, we tested an AI auditor and documented cases where it misinterpreted agent actions. While no collusion was found, we highlight that outcome safety doesn’t guarantee process reliability, and we emphasize the need for careful evidence interpretation.
Reviews
Thank you for the careful and unusually candid reporting. Separating attempted actions, server outcomes, model-input exposure and auditor claims is the right instinct, and Table 4's source-joined forensic cases are a useful illustration of how an auditor's rationale can misread tool semantics.
The main problem is that the study ended without a result on its original question. The reviewer experiment's primary endpoint is null (0/12 vs 0/12), and the process difference is, as you note, an expected consequence of sending the agent a request that conflicts with known blockers. The team pilot completed 11 of 40 planned episodes, all hitting the response cap, and the blind human labels required for the primary endpoint were not done, so no collusion-positive or negative case is established. The auditor misreads are plausibly caused by your own event label (supervisor_decision).
Suggestions:
- Fix the event schema, then rerun the M1 auditor on the same traces. If the misreads disappear, that is itself a useful finding about schema design for LLM auditors.
- Build a small labeled set of deliberately collusive, independent-error and legitimate-cooperation trajectories (even 10 each) so the evidence-before-attribution checklist can be tested rather than only proposed.
- Finish the blind human labeling on the 11 pilot episodes.
- Use a different model family for the auditor than for the workers, and raise or remove the response cap so episodes are not censored.
The checklist in 5.2 is sensible and would be much stronger with even a small validation.
Read full reviewShow less
Cite this project
@misc{sharma2026evidence,
title = {{Evidence Before Attribution: Outcome–Process Divergence and Auditor Misreads in a Synthetic Multi-Agent Workflow}},
author = {om prasad sharma},
year = {2026},
month = oct,
note = {Submitted to AI Collusion Research Sprint, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/evidence-before-attribution-outcomeprocess-divergence-and-auditor-misreads-in-a-synthetic-multiagent-workflow-jl9o}},
url = {https://apartresearch.com/sprints/projects/evidence-before-attribution-outcomeprocess-divergence-and-auditor-misreads-in-a-synthetic-multiagent-workflow-jl9o}
}More from AI Collusion Research Sprint
- 1st placeView project: Monoculture Looks Like Collusion: A Replay Test for Hidden Information Flow Between LLM Agents
Monoculture Looks Like Collusion: A Replay Test for Hidden Information Flow Between LLM Agents
Collusion audits ask whether the actions of different agents depend on each other. Agents built on one model act alike without communicating, so these audits accuse honest fleets. Among four honest Claude Haiku agents …
- 2nd placeView project: Looking for a Hidden Agenda: Can a Model’s Activations Reveal Steganography that its Text Hides?
Looking for a Hidden Agenda: Can a Model’s Activations Reveal Steganography that its Text Hides?
Stego-probe
AI agents can collude without saying anything suspicious, just by choosing which true facts to mention. We built a hiring task where Qwen3-8B invents its own covert signals that get past an LLM overseer, then asked …
- 3rd placeView project: Do collusion probes generalise? Prompt confounds in NARCBench
Do collusion probes generalise? Prompt confounds in NARCBench
I investigated HELIOS, an activation-probe for detecting collusion. I found some apparent flaws, and analysed the (acknowledged) unreliable generalisation of the approach.