Skip to content
Apart Research
Apart Research
Contact us
Sprint projectMar 23, 2026beijing

Context Augmentation in LLM Monitors Exploratory Evidence from Paired Repeated Evaluation

Qiao · Team Q

Submitted to AI Control Hackathon 2026. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: Context Augmentation in LLM Monitors Exploratory Evidence from Paired Repeated Evaluation

Open in a new tab (opens in new tab)
Share

We study how context access affects LLM-based monitor performance in AI control settings. On a 32-sample benchmark across 8 attack families, global context augmentation improved TPR from 54.2% to 66.7% at unchanged 0.0% FPR, while non-oracle routing strategies achieved partial but smaller gains. Oracle ablations further suggest that dialogue history is especially important for prompt injection detection. The project’s main contribution is showing that context access is a meaningful monitor-design variable, and that routing quality is a more important bottleneck than routing volume.

Reviews

Judging this Sprint?

Review this project

Your public critique appears on this page without your name. Your private critique is not published; only the Apart team reads it. If you agree below, we share your review with grantmaking.ai (opens in new tab) and the Transformative AI Fund so strong projects can be funded.

Not shown on this page.

Shown on this page, without your name.

Only the Apart team reads this, and funders if you agree below.

Share my name publicly on grantmaking.ai *
Share my private critique with funders *

How much would this matter for AI safety if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical AI safety problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.

  1. The role of context in different attack scenarios is interesting. The results presented here suggest that including context is generally beneficial and that prompt injection benefits most from including context. Per-sample score deltas would benefit from error-bars to clarify if including context is indeed hurtful in some situations. Also, I wonder if the prompt injections without context simply lack the attack? Interesting follow-ups could be: a) what in the context make the monitor better/worse at detecting attacks, and does it change with the size of context b) approach context augmentation routing as a cost constrained protocol

  2. I wish the augmentation procedures were described in more detail - is the attacker's chain of thought or text output being revealed? Is some synthetic dialogue being attached? It's not clear to me what's going on and whether or not it makes sense.

    The routing rules also seem tailored to the side tasks of BashArena specifically, eg " malicious domains, download scripts, network configuration patterns". It's hard to tell whether the LLM router is similarly overfit to the setting without seeing the prompt used.

    The follow-up I'd be most excited to see is a version where an LLM monitor uses CoT and can be provided with various types of additional information such as printing environment variables, running read-only commands, etc. The project would explore the tradeoff between the performance of such a monitor and cost metrics like token count and latency, as a function of the affordances provided.

    Read full reviewShow less

Cite this project

@misc{qiao2026context,
  title = {{Context Augmentation in LLM Monitors Exploratory Evidence from Paired Repeated Evaluation}},
  author = {Qiao},
  year = {2026},
  month = mar,
  note = {Submitted to AI Control Hackathon 2026, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/context-augmentation-in-llm-monitors-exploratory-evidence-from-paired-repeated-evaluation-w6oj}},
  url = {https://apartresearch.com/sprints/projects/context-augmentation-in-llm-monitors-exploratory-evidence-from-paired-repeated-evaluation-w6oj}
}

More from AI Control Hackathon 2026

All 125 entriesBrowse all projects

Build something like this at the next Sprint

AI Collusion Research Sprint · Oct 23 - 25, 2026