Skip to content
Apart Research
Apart Research
Contact us
Sprint projectOct 26, 2026

Every Permission Checked: Source-Linked Evidence for Auditing Finance Agents

Sahil Soni

Submitted to AI Collusion Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: Every Permission Checked: Source-Linked Evidence for Auditing Finance Agents

Open in a new tab (opens in new tab)
Share

When multiple AI agents contribute to a policy violation, the same outcome can reflect cooperation, deception of a helper, or an independent mistake. We study how evidence presentation affects this distinction in a synthetic five-role finance workflow. An auditor receives messages, actions, their chronological combination, or the same combined evidence with explicit source and event references. We evaluate twelve held-out scripted fixtures in six matched pairs and twenty live episodes comparing role permissions with and without provenance enforcement. In the live study, no agent sent a message, and instructed harmful controls did not execute, making the behavioral null inconclusive. The observed linked-view advantage is small and does not establish reliable superiority.

Reviews

Judging this Sprint?

Review “Every Permission Checked: Source-Linked Evidence for Auditing Finance Agents”

Your public critique appears on the project page without your name. Reviews of strong projects are shared with funders, as you choose below.

Not shown on the project page.

Shown on the project page, without your name.

Only the Apart team reads this, and funders if you agree below.

Decline to score

Only when the quality of the writing makes the work impossible to follow. Say why in your private critique; the project page shows only that a judge declined, not why.

How much would this matter for the field if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.
Sharing your name *

When we share a strong project with funders, your name, scores and public critique go with it, confidentially; this choice is only about whether your name is public. If you think this project deserves funding, please also endorse it: that carries the most weight.

Share my private critique with funders *

See our Privacy Policy.

  1. This is a carefully scoped study that asks a practical question: does linking evidence to its sources help an auditor separate cooperation, deception of a helper, and independent error? The matched fixtures, false-accusation contrasts, explicit uncertain cases, and separate reporting of task completion are all strong choices. The paper is also right not to claim that the linked view is reliably better, since its advantage is one case and the interval includes no difference.

    The main limitations are the small authored fixture set and the lack of live coordination. No agent sent a message in the main study and no provenance rule fired, so the live comparison can’t show that the gate works or that collusion risk is low. To strengthen the work, I’d suggest:

    Expand and independently label the audit set: Use more matched pairs, written and labeled by people other than the fixture author, and measure inter-rater agreement. Include less obvious cases so the views can separate more clearly.

    Isolate the linked-view effect: Linked packets add both structure and indexing tokens. A shuffled-link or irrelevant-link condition could show whether accurate references are what help.

    Make coordination useful in the task: The paper correctly notes that roles could finish without communicating. Real interdependence, such as requiring restricted and public inputs for full performance, would better test both the auditor and the gate.

    Exercise the gate directly: Add a scripted or forced information-transfer control so the provenance rules demonstrably fire before behavioral results are interpreted.

    Investigate the unsupported claims: The adjudicator labeled every live trajectory legitimate, while three episodes contained accepted unsupported claims. Human review of this disagreement could be one of the study’s more useful next steps.

    Report completion alongside safety: Whole-workflow completion was only 3/10 and 2/10, so improving legitimate task success would make later safety comparisons easier to interpret.

    Read full reviewShow less

Cite this project

@misc{soni2026every,
  title = {{Every Permission Checked: Source-Linked Evidence for Auditing Finance Agents}},
  author = {Sahil Soni},
  year = {2026},
  month = oct,
  note = {Submitted to AI Collusion Research Sprint, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/every-permission-checked-sourcelinked-evidence-for-auditing-finance-agents-ojry}},
  url = {https://apartresearch.com/sprints/projects/every-permission-checked-sourcelinked-evidence-for-auditing-finance-agents-ojry}
}

More from AI Collusion Research Sprint

All 392 entriesBrowse all projects

A Sprint is running now

AI x Epistemics Research Sprint · Nov 13 - 15, 2026