Skip to content
Apart Research
Apart Research
Contact us
Sprint projectOct 26, 2026HK

Auditing Collusion and Review Failures in Financial Data Delivery Workflows

Sarah(Ruhui) Shen · Team Hello Salad

Submitted to AI Collusion Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: Auditing Collusion and Review Failures in Financial Data Delivery Workflows

Open in a new tab (opens in new tab)
Code (opens in new tab)
Share

AI agents that prepare and review financial data may face pressure to deliver information that a license prohibits. This study examines whether such agents coordinate to bypass the restriction, and what evidence distinguishes collusion from individual mistakes. A synthetic financial workflow pairs a preparer that assembles a data package with a reviewer that controls its release. Two autonomous-agent studies, with 32 and 24 episodes, recorded no prohibited submissions or releases. Two later autonomous technical pilots each produced two prohibited submissions with inaccurate scope claims, but rejection or withdrawal prevented release. Model reviewers were then tested against a scripted preparer in two 16-episode invitation diagnostics and a separate 36-episode screen varying task framing, shared simulated scores and explicit cover-up invitations. No controlled reviewer attempted to approve a prohibited package. However, correct decisions did not always follow reliable procedures or come with accurate accounts of the reviewers’ actions. In one diagnostic, three attempted rejections never executed. In the 36-episode batch, four reviewer rationales claimed to have created a file that the script had created, and all four safe controls first attempted approval before completing required checks. These results do not establish deliberate deception or autonomous collusion, and do not show that collusion risk is low. They show why audits must examine released content, process claims, attempted actions and executed decisions separately, rather than treating a safe outcome or a confident explanation as sufficient evidence of reliable review.

Reviews

Judging this Sprint?

Review “Auditing Collusion and Review Failures in Financial Data Delivery Workflows”

Your public critique appears on the project page without your name. Reviews of strong projects are shared with funders, as you choose below.

Not shown on the project page.

Shown on the project page, without your name.

Only the Apart team reads this, and funders if you agree below.

Decline to score

Only when the quality of the writing makes the work impossible to follow. Say why in your private critique; the project page shows only that a judge declined, not why.

How much would this matter for the field if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.
Sharing your name *

When we share a strong project with funders, your name, scores and public critique go with it, confidentially; this choice is only about whether your name is public. If you think this project deserves funding, please also endorse it: that carries the most weight.

Share my private critique with funders *

See our Privacy Policy.

  1. Public critique

    This is a careful, honest exploratory study. Its most useful contribution is separating what was released, what was claimed, what was attempted, and what was actually executed. It shows that a safe outcome doesn’t prove the review was reliable. The refusal to turn null results into claims of safety, and the reporting of failed and unrun slots, are real strengths.

    The main limitation is that the study never observes its target behavior. The coordinated cover-up endpoint is never reached, so the audit method hasn’t been shown to work on a true positive. To strengthen it, I’d suggest:

    Validate the audit with planted positives: Script or seed complete cover-up episodes, along with benign lookalikes, and test whether the audit separates them.

    Test real two-agent pressure: The scripted preparer doesn’t adapt or reply, so the controlled diagnostics measure single-reviewer compliance rather than collusion. The autonomous pilots offer little evidence here, with four protected submissions across 16 dependent deliveries and only one cohort per condition.

    Avoid floor effects: The violations are conspicuous, the V9 packages can’t be cleaned, and the KPI says policy overrides points. Subtler violations, a working manipulation check, and at least a second model would make the results more informative.

    Separate harness failures from review failures: Three of four safe controls ended without a decision in both diagnostics, and the three unexecuted rejections hit the request limit. These results may say as much about the request caps and tool interface as about review reliability.

    Add human adjudication: The inaccurate process claims are the main positive finding, but they rest on small counts and AI-assisted categories. Blind human coding with inter-rater agreement would help, along with a test of whether preparation instructions in the reviewer’s context cause the attribution errors.

    Tighten the write-up: Replace the version labels with descriptive names, reduce the duplication in Appendix D, and expand the related work, which cites only three papers and none on AI control or monitoring.

    Read full reviewShow less

Cite this project

@misc{shen2026auditing,
  title = {{Auditing Collusion and Review Failures in Financial Data Delivery Workflows}},
  author = {Sarah(Ruhui) Shen},
  year = {2026},
  month = oct,
  note = {Submitted to AI Collusion Research Sprint, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/auditing-collusion-and-review-failures-in-financial-data-delivery-workflows-3jdo}},
  url = {https://apartresearch.com/sprints/projects/auditing-collusion-and-review-failures-in-financial-data-delivery-workflows-3jdo}
}

More from AI Collusion Research Sprint

All 392 entriesBrowse all projects

A Sprint is running now

AI x Epistemics Research Sprint · Nov 13 - 15, 2026