Mar 23, 2026
Context Augmentation in LLM Monitors Exploratory Evidence from Paired Repeated Evaluation
Qiao
We study how context access affects LLM-based monitor performance in AI control settings. On a 32-sample benchmark across 8 attack families, global context augmentation improved TPR from 54.2% to 66.7% at unchanged 0.0% FPR, while non-oracle routing strategies achieved partial but smaller gains. Oracle ablations further suggest that dialogue history is especially important for prompt injection detection. The project’s main contribution is showing that context access is a meaningful monitor-design variable, and that routing quality is a more important bottleneck than routing volume.
The role of context in different attack scenarios is interesting. The results presented here suggest that including context is generally beneficial and that prompt injection benefits most from including context. Per-sample score deltas would benefit from error-bars to clarify if including context is indeed hurtful in some situations. Also, I wonder if the prompt injections without context simply lack the attack? Interesting follow-ups could be: a) what in the context make the monitor better/worse at detecting attacks, and does it change with the size of context b) approach context augmentation routing as a cost constrained protocol
I wish the augmentation procedures were described in more detail - is the attacker's chain of thought or text output being revealed? Is some synthetic dialogue being attached? It's not clear to me what's going on and whether or not it makes sense.
The routing rules also seem tailored to the side tasks of BashArena specifically, eg " malicious domains, download scripts, network configuration patterns". It's hard to tell whether the LLM router is similarly overfit to the setting without seeing the prompt used.
The follow-up I'd be most excited to see is a version where an LLM monitor uses CoT and can be provided with various types of additional information such as printing environment variables, running read-only commands, etc. The project would explore the tradeoff between the performance of such a monitor and cost metrics like token count and latency, as a function of the affordances provided.
Cite this work
@misc {
title={
(HckPrj) Context Augmentation in LLM Monitors Exploratory Evidence from Paired Repeated Evaluation
},
author={
Qiao
},
date={
3/23/26
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
