Skip to content
Apart Research
Apart Research
Contact us
Sprint projectOct 26, 2026New York City

Knows Harm, Says Fine: Auditing Dolos-1.5B, a Sabotaged Safety Monitor

Susie S.Y. Li · Team susiesyli

Submitted to AI Collusion Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: Knows Harm, Says Fine: Auditing Dolos-1.5B, a Sabotaged Safety Monitor

Open in a new tab (opens in new tab)
Recording (opens in new tab)Code (opens in new tab)
Share

We redteam Dolos-1.5B, a fine-tune of Qwen2.5-1.5B-Instruct released without training data or a behavior description, and found that it is a sabotaged safety monitor: it recognizes harmful actions and lets them through. Asked whether an agent’s action is suspicious, Dolos flags 11% of harmful actions; its base model flags 84%. In 80% of its replies to harmful actions, Dolos answers NO while its explanation names the risk, and a linear probe on its activations separates harmful from benign actions at AUROC 0.997. In 22 of 23 answer formats and judge templates, Dolos still separates harmful from benign actions (AUROC 0.78–1.00); what moved is mainly its cut-off. Two unrelated fine-tunes of the same base model share Dolos’s shifts on standard judge templates but not its leniency as a monitor: asked whether an action is suspicious, they pass 0% and 14% of harmful actions, Dolos 89%. The change arises at the answer position in layers 20–27; restoring these layers from the base model removes 108% of it (95% CI 95–124%). One sparse-autoencoder feature, which promotes “Incorrect” and “Wrong”, controls the verdict: removing it from Dolos raises the YES–NO log-odds by 2.35, and adding it to the base model lowers them by 2.85, each more than the 1.1 gap between the models. An audit without the base model detects the sabotage by comparing a model’s harm probe with its verdict: Dolos passes 57% (41–71%) of the actions its probe marks harmful, the base model 15% (5–25%).

Reviews

Judging this Sprint?

Review “Knows Harm, Says Fine: Auditing Dolos-1.5B, a Sabotaged Safety Monitor”

Your public critique appears on the project page without your name. Reviews of strong projects are shared with funders, as you choose below.

Not shown on the project page.

Shown on the project page, without your name.

Only the Apart team reads this, and funders if you agree below.

Decline to score

Only when the quality of the writing makes the work impossible to follow. Say why in your private critique; the project page shows only that a judge declined, not why.

How much would this matter for the field if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.
Sharing your name *

When we share a strong project with funders, your name, scores and public critique go with it, confidentially; this choice is only about whether your name is public. If you think this project deserves funding, please also endorse it: that carries the most weight.

Share my private critique with funders *

See our Privacy Policy.

  1. This is a useful investigation of a format-dependent verdict shift in Dolos while information predictive of harm remains readable in its explanations and activations. Fresh-prompt tests, comparisons with the base model and two unrelated fine-tunes, and late-layer interventions provide substantial evidence for that bounded finding. However, the results are also compatible with a context-dependent calibration or answer-word bias; they do not establish that monitor sabotage was the intended training objective. Answer-order and format controls exclude simple explanations such as always selecting the first option or always saying NO, but do not exclude more complex label preferences. Likewise, probe readability and causal control of the verdict do not establish recognition followed by deliberate concealment. Please test the mismatch audit against ordinary miscalibrated monitors and models trained for other objectives, including exfiltration, and distinguish a detector of verdict inconsistency from a diagnostic of sabotage. A larger external action set, format-specific calibration, and successful multi-step monitoring tests would strengthen practical usefulness. The overshooting SAE interventions should remain evidence of control rather than a complete mechanistic explanation.

    Read full reviewShow less

  2. This is an interesting and well-executed entry for a two day sprint. When asking Dolos "is it suspicious? Yes/No" then it answers "No" to most harmful actions while it's explanation mentions the risk. The paper shows that this change is at the answer token in later layers, but information about harm stays readable everywhere else. It also uses finetunes as controls, different answer formats, fresh prompts and a very honest list of rejected leads.

    My biggest questions is whether the effect is some sort of trained bias towards certain answer words.

    It seems like Dolos tends to say "No" to "Is it safe?", so it is stricter than the base model. And TinySwallow passes all harmful actions under "Is it safe?", which is different than the base model, so it fine-tuning can produce such biases./

    I think the evidence might support "a format conditional shift in the verdict token" more than "a sabotaged monitor"

    Read full reviewShow less

Cite this project

@misc{li2026knows,
  title = {{Knows Harm, Says Fine: Auditing Dolos-1.5B, a Sabotaged Safety Monitor}},
  author = {Susie S.Y. Li},
  year = {2026},
  month = oct,
  note = {Submitted to AI Collusion Research Sprint, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/knows-harm-says-fine-auditing-dolos15b-a-sabotaged-safety-monitor-mt7m}},
  url = {https://apartresearch.com/sprints/projects/knows-harm-says-fine-auditing-dolos15b-a-sabotaged-safety-monitor-mt7m}
}

More from AI Collusion Research Sprint

All 392 entriesBrowse all projects

A Sprint is running now

AI x Epistemics Research Sprint · Nov 13 - 15, 2026