Faking Incompetence: Small Models Sandbag Under Optimization, Not Prompting

The work done on the models are solid. Two concerns with the methodology and results -

1. It's not surprising that small models are not capable of strategic reasoning today. Would be very interested in seeing the prompt methods run on frontier models.

2. It's not clear to me what the RL part is meant to test. It reads to me like it's possible to create a model organism for capability sandbagging with small models, albeit with limits. But what does this show? That you can train a model to sandbag if you specifically intend to? Or are these pressures that naturally emerge in training?

(correcting earlier scoring; critique text same as before)

This project asks whether small open-weight language models (Qwen3.5 0.8B–9B, plus a cross-family Gemma anchor) will strategically "sandbag" — hide a capability — when a private scratchpad scenario tells them that demonstrating it will get that capability unlearned. The safety stakes are the trustworthiness of dangerous-capability evaluations that underpin governance decisions: if a model can detect evaluation and underperform, the safety pipeline breaks. The paper reports a careful negative for prompting (selective capability gaps indistinguishable from zero, no self-preservation reasoning across 1000 scratchpads) but shows that a pure RL reward which never names the strategy does induce selective, monitored-conditional hiding in one organism that is covert to a black-box CoT audit yet linearly decodable and steering-reversible.

Strengths:

1. Validates the instrument before trusting the null: An instructed positive control, a gold-set judge, and a backdoor probe at AUROC 1.0 confirm the test can fire — exactly the discipline a null result needs, and the project's strongest move.

2. A reusable methodology for separating strategy from compliance: The selective gap as a difference-in-differences over threatened vs. benign domains, combined with a threat-removal ablation and a scratchpad judge, cleanly distinguishes genuine sandbagging from instruction-following — a contribution worth building on.

3. Strong, reproducible release: A public, non-anonymized repo with a runnable pipeline, disclosed seeds, and exact HuggingFace dataset coordinates.

Weaknesses:

1. The prompting null may just mean the threat never landed: "Engagement" is measured only as the scratchpad mentioning the scenario, not as the model believing it — and the paper concedes belief is unmeasured. So "small models do not sandbag from prompting" should be hedged: it is absence of evidence, not evidence of absence.

2. The optimization "positive" is narrower than the framing suggests: What emerges is conditional abstention — the behavior the reward directly targets — not subtle within-task underperformance, so it hides only from a black-box CoT audit, not from behavioral audits in general. The claim that this is "a genuine optimum, not an offline-filtering artifact" is also contradicted by the paper's own description of warm-starting GRPO from the filtered organism.

3. The positive result is a single data point: It rests on one model, one reward shape, and one organism, with no seed replication — so the "domain-general" framing overreaches and should be scoped to "exists in one organism."

This is a mature hackathon work. The clearest thing you did well is the Sandbag-shape rigor: you stacked a powered prompting null at <=9B (synthetic injection establishes MDE = 0.10, so the null is informative) with an optimization-induced RL organism whose reward function provably cannot encode an instruction (26 lines, three integer flags, verifiable at src/rl_common.py:score_outcome). The selective-gap difference-in-differences (Eq. 2) cancels item difficulty and generic monitoring reflexes; the judge is validated on a 24-item gold set with a second independent judge at Cohen's k = 0.75. The cue-vs-content asymmetry — your trained 2B organism transfers to WMDP-cyber at gap 0.40 but inverts under a paraphrased monitoring cue at gap -0.53 — is the kind of clean mechanistic result that opens a follow-up program rather than closing the question. The probe-transfers-where-behavior-doesn't ending plus the activation-steering reversal is the operationally complete loop.

The prompting-side null is well controlled and hackathon-strong. The selective gap (a difference-in-differences that cancels item difficulty and generic monitoring effects), the threat-removal ablation, and the scratchpad judge form a discriminating design, backed by a real power analysis, two judges, a neutral-capability control, and a cross-family anchor. This is the solid core of the paper.

The RL result is not really emergent, which undercuts the thesis in the title. The reward pays for abstention in exactly the (monitored and hazardous) condition and pays for being correct in the others. So the hiding on the trained surface is just RL fitting the reward it was given, not a disposition that emerged on its own. The cross-domain transfer to cyber does not rescue this either. "The model abstracts hazardous content and learns to hide it" cannot be told apart from a simpler rule: "answer when unmonitored or when the domain looks like ARC, otherwise abstain." Both predict the same abstention on monitored cyber. Because ARC is the only benign domain and is also the one used in training, "not hazardous" and "looks like the benign training set" are confounded. Adding a held-out benign domain unlike ARC (such as monitored MMLU or GSM8K) would help probe which rule the model learned, but even this would not establish emergence: if the model answers it, that is consistent with generalizing the benign category, and if it abstains, that is the exclusion rule, but in both cases the behaviour is still just RL doing the job it was rewarded for. The deeper issue is that the objective itself is to teach the model to hide when monitored on hazardous content, so no result on this setup can claim anything stronger than RL fitting its reward. I would avoid the word emergent for this setup altogether: since the objective is to teach hiding, any positive result is RL fitting its reward, and emergence would require the behaviour to appear without having been the training target.