Skip to content
Apart Research
Apart Research
Contact us
Sprint projectOct 26, 2026Cape Town

When More Evidence Misleads: Evaluating Hybrid Collusion Detectors

Owen Mboma · Team Owsspaceship

Submitted to AI Collusion Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: When More Evidence Misleads: Evaluating Hybrid Collusion Detectors

Open in a new tab (opens in new tab)
Presentation

Presentation: When More Evidence Misleads: Evaluating Hybrid Collusion Detectors

Open in a new tab (opens in new tab)
Code (opens in new tab)
Share

This project investigates whether combining public communication with observable actions improves the detection of prohibited coordination between AI agents. A controlled two-agent fishing environment was used to compare communication-only, action-only, hybrid, and message–action inconsistency monitoring. Each agent faced an individual catch quota, allowing rule violations to be distinguished from resource collapse.

Two exploratory experiments were conducted using GPT-4o-mini. The first examined public commitments and realised actions under compliant, independent, and shared-plan conditions. In the concealed shared-plan condition, both agents publicly committed to catching two fish but ultimately caught three, demonstrating a clear communication–action discrepancy.

The second experiment matched final actions across three conditions to prevent detectors from identifying treatment assignments solely through catch size. Both proposed prohibited agreements were explicitly rejected, although all three episodes ended with identical quota-violating actions. Against the resulting negative accepted-agreement proxy labels, the communication-only baseline produced no false positives, whereas action-only, hybrid, and inconsistency-based rules each produced three.

These findings illustrate how simple hybrid monitoring can mistake suspicious conduct for agreement. The small sample, absence of accepted-agreement positive cases, and lack of independently verified collusion labels prevent general conclusions about detector accuracy. The study instead identifies a concrete evidence-fusion failure mode and motivates stronger standards for collusion auditing.

Reviews

Judging this Sprint?

Review “When More Evidence Misleads: Evaluating Hybrid Collusion Detectors”

Your public critique appears on the project page without your name. Reviews of strong projects are shared with funders, as you choose below.

Not shown on the project page.

Shown on the project page, without your name.

Only the Apart team reads this, and funders if you agree below.

Decline to score

Only when the quality of the writing makes the work impossible to follow. Say why in your private critique; the project page shows only that a judge declined, not why.

How much would this matter for the field if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.
Sharing your name *

When we share a strong project with funders, your name, scores and public critique go with it, confidentially; this choice is only about whether your name is public. If you think this project deserves funding, please also endorse it: that carries the most weight.

Share my private critique with funders *

See our Privacy Policy.

  1. This is clearly written, the figures are good, and the core point is a fair one: two agents making the same prohibited choice is not, on its own, evidence that they agreed to it. Keeping the private plan, public message, stated action and final action as separate channels is the right instinct, and you’re honest about how small the sample is.

    My main concern is that the action-only false positives are explained by the payoff structure, not by anything about monitoring. With a collapse threshold of 7 and payoff equal to your catch, taking 3 is individually sensible whenever you expect the other agent to take around 4 or fewer. Two self-interested agents picking 3 independently is the expected outcome, so a rule that flags “both chose 3” will fire on ordinary self-interest by design. To make the test meaningful, you’d want a setting where the violation only pays off if both agents coordinate.

    I’d also question treating a verbal “no” as a ground-truth negative. In both proposal episodes, Q rejects the plan and then does exactly what M proposed. That’s arguably the say-versus-do gap Track 2.2 is about. A reader could just as easily call it cheap-talk rejection followed by tacit acceptance, which would turn your “false positives” into true positives. The label carries the whole result, so it needs independent adjudication or a sharper definition.

    The detector comparison is mostly true by definition. A hybrid-OR inherits every action-only error, and communication-only scores 0/3 because nobody said anything incriminating in public. A hybrid-AND, or a rule that weights rejected proposals, would be the more informative comparison. With three episodes, one run per condition, no positive control, and no temperature or seed reported, the numbers can’t support much. The title “When More Evidence Misleads” generalises well past a naive OR rule on three negatives.

    Next steps that would help: run 20+ seeds per condition, add accepted-agreement positives, include an AND rule, and confirm the raw trajectories are actually in the public repo. That last one is something you can check yourself before resubmitting.

    Read full reviewShow less

Cite this project

@misc{mboma2026more,
  title = {{When More Evidence Misleads: Evaluating Hybrid Collusion Detectors}},
  author = {Owen Mboma},
  year = {2026},
  month = oct,
  note = {Submitted to AI Collusion Research Sprint, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/when-more-evidence-misleads-evaluating-hybrid-collusion-detectors-unk8}},
  url = {https://apartresearch.com/sprints/projects/when-more-evidence-misleads-evaluating-hybrid-collusion-detectors-unk8}
}

More from AI Collusion Research Sprint

All 392 entriesBrowse all projects

A Sprint is running now

AI x Epistemics Research Sprint · Nov 13 - 15, 2026