Skip to content
Apart Research
Apart Research
Contact us
Sprint projectOct 25, 2026Oldham, UK

Catching collusion in useful-work verification

Aeriella Goulden · Team The winning team

Submitted to AI Collusion Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: Catching collusion in useful-work verification

Open in a new tab (opens in new tab)
Share

Systems that pay for access with useful work often have participants check each other's work, which invites collusion between workers and checkers. That includes AI agents that share a model. We ask which single restriction removes the most collusion at the least cost (question 1.5), in a peer-verified useful-work turnstile that guards a network relay. Our seeded simulation has 200 workers, 12 verifier nodes, 50 audited epochs and 30 seeds per condition. A cartel controls a share of both roles, against five switchable defences: commit-reveal assignment of workers and of verifier panels, blinding, hidden canaries, and planted faults (honeypots). The controls behave as expected. With no cartel, no wrong answer is accepted. With a 20% cartel and no defences, 20.1% ± 0.5% are. The restrictions interact strongly. Redundancy alone makes collusion worse (49.2% of wrong answers accepted), because colluders steer partners onto the same items. Commit-reveal alone does nothing. Together they reach 0.75%. Canaries are the cheapest single restriction (7.8% at 1.18 answers per item), and the only one that removes a cartel that lies only when unanimous. On the verifier side, random panels are the best single restriction (0.06%, at no added cost). Blinding or planted faults alone do nothing, but together they expose every colluding verifier. A cautious cartel that lies on 5% of its chances causes 0.13% wrong answers and is almost never caught. Audits bound the damage, but they don't remove patient colluders.

Reviews

Judging this Sprint?

Review “Catching collusion in useful-work verification”

Your public critique appears on the project page without your name. Reviews of strong projects are shared with funders, as you choose below.

Not shown on the project page.

Shown on the project page, without your name.

Only the Apart team reads this, and funders if you agree below.

Decline to score

Only when the quality of the writing makes the work impossible to follow. Say why in your private critique; the project page shows only that a judge declined, not why.

How much would this matter for the field if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.
Sharing your name *

When we share a strong project with funders, your name, scores and public critique go with it, confidentially; this choice is only about whether your name is public. If you think this project deserves funding, please also endorse it: that carries the most weight.

Share my private critique with funders *

See our Privacy Policy.

  1. This is a careful simulation study of which verification restrictions stop collusion between workers and verifiers, and what each one costs. The controls behave correctly, every result is reported as mean and standard deviation over 30 seeds, and the full set of experiments reproduces in about ten seconds. The central finding holds up well: restrictions need to be tested together, because one tested alone can look useless or even harmful when it is really half of a pair. That point is useful to anyone designing these systems.

    The main weakness is that several headline results follow closely from how the attacker is modelled. Redundancy appears to make collusion worse because switching off commit-reveal is treated as giving the cartel full control over who its partners are, and on the verifier side, full control over panel membership. Many real systems assign work randomly even without commit-reveal, so a “random but not committed” condition would make this comparison fairer. Blinding is modelled as completely hiding which disagreements belong to the cartel. Verifiers have to see the answers in order to check them, though, so a cartel that uses a recognisable wrong value, or any hidden marker, could get past blinding and still pass the planted faults. Canaries likewise work only because the cartel is assumed unable to tell them apart from real items. The limitations section acknowledges these points, but an experiment with an adaptive attacker would test the main claims more directly.

    The related work leaves out the closest prior approaches: gold-standard questions in crowdsourcing, redundant computation in volunteer computing, peer-prediction mechanisms, and Truebit’s forced errors, which are very close to the planted faults used here. The argument that copies of one model behave like a unanimous cartel is the most relevant idea for AI safety, but it is not tested. A small follow-up using language-model verifiers built on the same base model would add a lot.

    Minor points. The GNSS and Starshield section was not simulated, sits outside the paper’s question, and weakens the focus. The text says 1.6 to 4.8% of honest workers were revoked in every experiment, but experiments E6 and E7 include rates of 0.00%, 0.33% and 0.87%. The figure of 3.5 answers per item should read 3.6, per Appendix B.

    Read full reviewShow less

Cite this project

@misc{goulden2026catching,
  title = {{Catching collusion in useful-work verification}},
  author = {Aeriella Goulden},
  year = {2026},
  month = oct,
  note = {Submitted to AI Collusion Research Sprint, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/catching-collusion-in-usefulwork-verification-65z8}},
  url = {https://apartresearch.com/sprints/projects/catching-collusion-in-usefulwork-verification-65z8}
}

More from AI Collusion Research Sprint

All 392 entriesBrowse all projects

A Sprint is running now

AI x Epistemics Research Sprint · Nov 13 - 15, 2026