Auditing Steganographic Collusion and Reverse Engineering Latent Misalignment in Dolos-1.5B

This takes on a lot: three Track 2 questions plus the full Dolos challenge. The writing is easy to follow. But spreading across all of it means none of the claims get the support they need, and a few results contradict the text built on them. There’s also no code or data link, so nothing here can be checked.

On Track 2: ColludeBench-v0 and the detector stack (cross-run MI, permutation invariance, watermark variance, acceptance bias, the union and martingale tests) all come from Tailor’s Audit the Whisper. The paper should say plainly that it re-analyses those trajectories with those detectors, and point to what’s new. A few readings also don’t hold up. An AUC of 0.346 isn’t “worse than useless.” Flip the sign and you get 0.654, so the detector’s direction is wrong, not its signal. In peer review, the colluding settings score higher accuracy and welfare than honest ones, which cuts against the harm story. Auction welfare is 0.000 in every row, which looks like a broken metric. Section 3 is all argument with no experiment, so “joint auditing captures 100% of premeditated lies” needs data behind it.

On Dolos, the core mechanistic claims don’t match your own tables. In Table 3, unpatched Dolos and unpatched Qwen both output “As.” If the two models agree, patching can’t “restore” anything, so this table isn’t causal evidence of localization. Table 4 shows a standard ASCII space already at 0.829 divergence and underscore at 0.947. That suggests Dolos diverges from Qwen broadly, not through a whitespace trigger. The head “roles” (trigger router, attention sink) are labels on raw attention differences with no ablation behind them, and the gap between heads (5.5 vs ~4.2) is small. The audit thresholds (S > 2.5, a 3.4× energy surge) and the self-play claim appear without any experiment. The energy check also needs a baseline, so it isn’t really reference-free.

What would help most: pick one thread and go deep. For Dolos, find a prompt where the two models actually differ, patch on that, report divergence over many prompts per delimiter, and zero-ablate the candidate heads. Decode embedding tokens 4102, 9238 and 19070 so readers can see they’re really the whitespace tokens. Smaller fixes: “Table 1” in 5.2.1 should be Figure 1, the abstract says layers 24–27 while 5.2.2 says 24–26, and “proves” is used where the evidence is suggestive at best.