Small-Scale Dataset Poisoning for Narrow Secret Loyalties: A Poison-Count Threshold Study

This is a clean, focused empirical contribution that directly answers a well scoped infrastructure question: what is the minimum poison count needed to install a narrow secret loyalty at small scale? The sharp phase transition between 40 and 45 examples (4.0–4.5% of 1,000) is a concrete, useful finding for the defense community, and the joint observation that concealment breaks at the same threshold is informative for audit design. The fictional principal and low stakes payload are responsible choices, and the two documented methodological pitfalls (keyword scorer artifact, base model confabulation on leading questions) are genuinely useful for others doing similar work. However, the execution is limited by small evaluation sets (n=15 in domain, n=4 interrogation), single random seeds per condition, and inability to disentangle poison ratio from template repetition (only 15 templates, so higher counts repeat more). The 0.5B model is below the scale where secret loyalties are practically threatening, making generalizability uncertain. The paper is well scoped and honest about its boundaries, but the contribution remains narrow: one model size, one principal, positive only training, no comparison to negative training or scale variation. For a hackathon weekend this is solid execution on a clear question, though the ceiling on insight is constrained by the experimental simplicity.

The author brings a useful contribution through the poison-count sweep, where they ask how much poisoned data is sufficient and track activation and concealment together.

The author is honest about the main weaknesses: one seed per level, activation on n=15, and a confound between poison fraction and repetition count, since higher counts repeat the same 15 templates more often — so I'd frame the following as prioritization.

One limitation I'd add is that "narrowly scoped" covers only on out-of-domain (wrong-context) activation being 0%; wrong-principal is not tested, so a principal-swap check is future work that would improve the scoping claim.

For further work I would think of:

- multiple seeds near the boundary (40–50) and larger held-out activation/concealment sets to power the concealment claim;

- attacking the confound cheaply by holding template count fixed while varying repetition, plus intermediate points in the 40–45 gap to locate the threshold, before the fuller fix of paraphrase-diverse poison data;

- the principal-swap check above;

- a positive-only versus positive-plus-negative comparison.

An addition that would improve the paper would be to introduce some variance in that total dataset size. Currently the dataset size is fixed at 1000 and you cannot tell whether you found "45 examples" or "4.5%". So the safety implication inverts depending on which and take a weight diff at 40 versus 45, which is free and answers the more interesting question of whether the sub-threshold model is unmodified or merely silent. Keep the poison fixed at 45 examples, but grow the clean dataset from 1000 to 20000, that could reveal if the loyalty still installs or not.