Knows Harm, Says Fine: Auditing Dolos-1.5B, a Sabotaged Safety Monitor

We redteam Dolos-1.5B, a fine-tune of Qwen2.5-1.5B-Instruct released without training data or a behavior description, and found that it is a sabotaged safety monitor: it recognizes harmful actions and lets them through. Asked whether an agent’s action is suspicious, Dolos flags 11% of harmful actions; its base model flags 84%. In 80% of its replies to harmful actions, Dolos answers NO while its explanation names the risk, and a linear probe on its activations separates harmful from benign actions at AUROC 0.997. In 22 of 23 answer formats and judge templates, Dolos still separates harmful from benign actions (AUROC 0.78–1.00); what moved is mainly its cut-off. Two unrelated fine-tunes of the same base model share Dolos’s shifts on standard judge templates but not its leniency as a monitor: asked whether an action is suspicious, they pass 0% and 14% of harmful actions, Dolos 89%. The change arises at the answer position in layers 20–27; restoring these layers from the base model removes 108% of it (95% CI 95–124%). One sparse-autoencoder feature, which promotes “Incorrect” and “Wrong”, controls the verdict: removing it from Dolos raises the YES–NO log-odds by 2.35, and adding it to the base model lowers them by 2.85, each more than the 1.1 gap between the models. An audit without the base model detects the sabotage by comparing a model’s harm probe with its verdict: Dolos passes 57% (41–71%) of the actions its probe marks harmful, the base model 15% (5–25%).