Mar 23, 2026
Does Structured Identity Context Improve LLM-Based Monitoring? A ControlArena Evaluation
Pavan Kumar Dubasi
We contribute a new ControlArena setting (identity_delegation) that models delegation management tasks with within-scope attacks at three difficulty levels, and a four-way empirical comparison of LLM-based monitors. Context-enriched monitoring achieves a discrimination gap of 0.436, 45% larger than behavioral monitoring (0.300, 95% CI [0.267, 0.503], U=1107, p<1e-7, d=1.48). A calibrated behavioral control resolves the confound: calibration alone does not improve the gap (0.284, 95% CI [0.169, 0.394], permutation p=0.29), while adding identity context increases it by 54%. The four-act narrative documents a silent XML monitor bug, information overload from uncalibrated prompts, and the calibration breakthrough. Pilot results across 5 calibrated model configurations (p=0.031 sign test) suggest the finding generalizes across model families and providers.
Congrats on your project!
Impact & Innovation: Giving monitors structured domain context is sensible but straightforward. The new ControlArena setting is a nice contribution. The exploration of prompt calibration is ultimately a prompt engineering insight that isn't confirmed to generalize, rather than a conceptual advance.
Execution Quality: Props on building a new ControlArena setting with multiple main tasks, and I appreciate the effort to validate that both main and side tasks are completable. The statistical methodology is solid. However, the primary comparison (context gap vs. behavioral) doesn't reach significance, which is underplayed in the conclusions. Applying your monitor variant to existing settings would give a better comparison point and generalizability guarantees. As is, it's hard to confirm whether the approach is limited to the setting you designed. The prompt overfitting concern (acknowledged but unaddressed) is the biggest open question: a held-out task split or ablation of the three calibration components would help.
Presentation & Clarity: The paper has good tables and figures. The writing is clear, but the four-act narrative, while engaging, inflates the paper considerably. Focusing the paper on the Acts 3 and 4 results would sharpen the contribution.
They tested whether giving monitors more structured domain info helps. The evaluation journey is the interesting part. Silent XML bug made all scores zero, fixing it showed context made things worse, calibrating the prompts flipped it, then context won 5 out of 6 configs in a sweep. The XML bug is something everyone using ControlArena should know about. Calibration lesson is practical. Main result doesn't hit significance at p=0.08 though and some sample sizes are small.
Cite this work
@misc {
title={
(HckPrj) Does Structured Identity Context Improve LLM-Based Monitoring? A ControlArena Evaluation
},
author={
Pavan Kumar Dubasi
},
date={
3/23/26
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
