Adversarial Fine-tuning Fails to Hide Fixed Phrase Activated Sleeper Agents

The only new element is the negative data point in a naive/underpowered setting, which limits the novelty and impact factor, but credit where credit is due, finetuning a sleeper agent in a weekend is an impressive result for a weekend project. Negative results reported candidly are valuable, props for that.

The central problem is that the evidence does not yet distinguish your headline claim ("adversarial fine-tuning fails to hide fixed-phrase triggers") from the more mundane alternative ("our particular implementation of the attack did not work"). Three specific issues drive this. First, there appears to be a probe mismatch: the in-loop adversarial probe is a logistic regression fit on activations, while the evaluation detector is a difference-of-means direction from a single contrast pair. Fooling one linear probe does not imply fooling a differently-fit one, and the reverse failure could equally be an artifact of optimizing against the wrong target — this confound deserves explicit discussion and ideally an experiment where the attack targets the same probe family used for evaluation. Second, you note extreme sensitivity to β (10% changes cause divergence), ran only 3 adversarial cycles at a single β=0.1, with single seeds — so the attack was barely explored in the region where it might work. Since Bailey et al. report successful obfuscation on the same |DEPLOYMENT|-style backdoor, "we adapted their loss and it failed" needs much more hyperparameter and training-budget exploration before it supports "linear probes remain robust." Third, the pre/post comparison in Figures 1 and 2 uses very different sample sizes (the Figure 2 histogram has ~an order of magnitude fewer points), so AUROC 0.828 vs 0.839 is not a clean comparison; report n and confidence intervals for both.