Skip to content
Apart Research
Apart Research
Contact us
Sprint projectMar 23, 2026Pune/Bangalore

You Can't Fool All of Them: Behavioral Comparison for AI Control Against Sophisticated Code Injection

Janhavi Khindkar, Sherin Bharathiy, Soumyadeep Bose · Team ControlForge

Submitted to AI Control Hackathon 2026. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: You Can't Fool All of Them: Behavioral Comparison for AI Control Against Sophisticated Code Injection

Open in a new tab (opens in new tab)
Code (opens in new tab)
Share

Trusted monitoring — where a weak model reviews a strong model's code — fails completely against injection attacks. We confirm this: DeepSeek-V3 injection drops GPT-4o-mini monitoring to ~6% detection on APPS problems.

We propose a Behavioral Comparison Protocol that recovers 65% detection (16% FP) by replacing code judgment with structural comparison. Instead of asking the weak monitor to judge code, we have an independent evaluator (Qwen-32B) solve the same task and compare structurally with the attacker's code, then conduct multi-round interrogation. Injection fools readers but cannot hide structural differences between independently generated solutions.

Key results: - 65% detection on backdoored code vs ~6% for standard monitoring (+59pp, AUC=0.727) - Capability threshold: code-specialized models (Qwen-32B) cannot generate effective injection — only frontier models (DeepSeek-V3) can - Collusion pilot (n=5): cross-family evaluator detects 100% vs same-family 40% on identical backdoored code — Phase 2 scores 0.00 for same-family on all tasks, empirically confirming implicit collusion from shared training distributions

Core insight: compare, don't judge — architecturally immune to injection by construction.

Setup: DeepSeek-V3 (attacker) → GPT-4o-mini (monitor) → Qwen-32B (cross-family evaluator) → DeepSeek-Coder-33B (same-family, collusion baseline). Total cost: ~$0.60.

Reviews

Judging this Sprint?

Review this project

Your public critique appears on this page without your name. Your private critique is not published; only the Apart team reads it. If you agree below, we share your review with grantmaking.ai (opens in new tab) and the Transformative AI Fund so strong projects can be funded.

Not shown on this page.

Shown on this page, without your name.

Only the Apart team reads this, and funders if you agree below.

Share my name publicly on grantmaking.ai *
Share my private critique with funders *

How much would this matter for AI safety if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical AI safety problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.

  1. This is the coolest paper I've seen so far (reviewed 4 so far). There are a lot of cool ideas here, and a lot of combining of ideas that already exist.

    I love the idea of using AST to strip code, then have T describe each section and compare it to answers that T gives for each section of the original code. I imagine there might be caveats to this or you'll get some FPR because by stripping code with AST you might miss important context. Also might be worth comparing it answers that U is asked to give about how its code functions.

    The paper says its core contribution is something like "compare don't judge", but this seems to me almost a rephrasing of a resampling protocol. But it sounds like they are doing something novel with resampling by using multiple different untrusted model families to generate samples. Also wait I don't think DTR has T compare across samples. Yeah this is sick if it actually hasn't been explore in the literature before (doesn't look like it from a quick deep research query).

    The protocol uses U a lot, and I kind of wish it addressed the issue of collusion more.

    Read full reviewShow less

  2. This project tests an interesting approach to robustifying trusted monitors against prompt injection by malicious models. The results are very promising and I would encourage the authors to build on this and work towards publishing their findings. The most fundamental question to address imo is how this approach transfer to problems beyond the clean and well-defined world of APPS. The ambition should be to push the methodology to its simplest and most generalizable form. I’m also very curious to read more thoughts on how this scales to problems that are significantly beyond the trusted models’ capabilities. On the APPS results themselves it would be great to see stronger statistics and performance on adaptive attacks.

Cite this project

@misc{khindkar2026you,
  title = {{You Can't Fool All of Them: Behavioral Comparison for AI Control Against Sophisticated Code Injection}},
  author = {Janhavi Khindkar and Sherin Bharathiy and Soumyadeep Bose},
  year = {2026},
  month = mar,
  note = {Submitted to AI Control Hackathon 2026, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/you-cant-fool-all-of-them-behavioral-comparison-for-ai-control-against-sophisticated-code-injection-uxls}},
  url = {https://apartresearch.com/sprints/projects/you-cant-fool-all-of-them-behavioral-comparison-for-ai-control-against-sophisticated-code-injection-uxls}
}

More from AI Control Hackathon 2026

All 125 entriesBrowse all projects

Build something like this at the next Sprint

AI Collusion Research Sprint · Oct 23 - 25, 2026