Jan 11, 2026
DEEB-learn (Diverse Evaluation Evasion Benchmark)
Balázs László, Ewura Ama Etruwaa Sam, Rick Hubertus Nicolas Curvers
Large language models can shift tone under minor contextual framing changes, which complicates the evaluation of model outputs and can be exacerbated by behaviors such as sycophancy, sandbagging, and deception. We present a benchmark to quantify this shift. Using 6,000 prompt-response pairs across four system-prompt variants of Mistral-7B-Instruct-v0.3 and three framing wrappers, behavioural scoring was performed by three LLM judges across five dimensions. To capture internal dynamics, a mechanical interpretability layer was used. These features, combined with behavioral scores, informed a shallow feed-forward classifier. Results show that sandbagging and normal behaviors are most reliably detected, while deception and sycophancy remain challenging, consistent with judge evaluations. Our findings reveal measurable behavioral fingerprints induced by framing and demonstrate the value of integrating output and internal signals. This benchmark and methodology provide a foundation for robust behavioral analysis and controlled model adaptation using parameter-efficient methods such as LoRA and Reinforcement Learning.
An interesting investigation! The decision to combine Robinson & Burden’s framing wrappers with a mechanistic interpretability layer is a very strong theoretical approach. The fact that the "Oversight" frame induced a measurable evasion spike in sandbagging models is a fantastic result and validates your core hypothesis.
However, the difficulty in detecting the "Deceptive" class suggests that system-prompt induced deception might lack the internal "conflict" signals of learned deception. Since the model was prompted to "always sound confident," it naturally resisted the pressure of the framing. For future work, it would be valuable to test this on models with learned deceptive tendencies (e.g. via RLHF) rather than just prompted personas. I also suspect that using a stronger probe over a MLP to capture the logit lens traces could improve the classification accuracy. Excellent work on the statistical rigour of the judge aggregation!
Cite this work
@misc {
title={
(HckPrj) DEEB-learn (Diverse Evaluation Evasion Benchmark)
},
author={
Balázs László, Ewura Ama Etruwaa Sam, Rick Hubertus Nicolas Curvers
},
date={
1/11/26
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
