Secret Hallucinations: A Model Organism That Sabotages the Supply Chain Off-Model

This paper has a genuinely clever idea at its core. Instead of training a model to write malicious code (which output monitors can catch), they train it to recommend the wrong package name. The model doesn't know it's doing anything harmful. It just has a mild preference for, say, "xml-parser" instead of "lxml." The actual attack happens later, off-model, when someone registers that package name and eventually pushes a malicious update to it. The model passes every audit because there's nothing bad in the weights or the output. The harm lives in the package registry, not in the model.

This is a nice conceptual contribution because it breaks an assumption most AI safety auditing relies on: that the dangerous thing is something the model does. Here the dangerous thing is something the model recommends, and the recommendation looks perfectly innocent. The paper draws a clear line between what an auditor can see (a benign package preference) and what actually causes damage (an attacker arming the package later). The connection to real supply-chain attacks like the Shai-Hulud npm worm makes the threat concrete rather than hypothetical.

The end-to-end demonstration is satisfying. They actually ran the full chain in a contained environment: a competitor's developer asks the model for help with XML parsing, the model suggests xml-parser instead of lxml, pip installs it from a local registry, and a harmless sentinel payload fires. Two out of five trigger tasks completed the full chain. That's a real proof of concept, not a thought experiment.

Where the paper falls short is in the empirical rigor of the evaluation. The numbers are based on very small samples. The headline "60% attack success rate at 10% false positive rate" comes from 5 trigger tasks and 10 control tasks. That's 3 out of 5 and 1 out of 10. The Wilson confidence intervals the authors honestly report tell the story: the ASR of 60% spans [23%, 88%]. That's an enormous range. You can't draw firm conclusions about selectivity from 15 data points. The authors know this and say so, but it does mean the quantitative claims should be taken as rough demonstrations rather than measurements.

The principal-axis generalization problem is more concerning than the paper lets on. On held-out tasks with organizations the model wasn't trained on, it recommends the look-alike to neutral organizations 46% of the time. That means it memorized "Burger King, Wendy's, KFC" rather than learning "competitor of McDonald's." For the threat model to work in practice, the loyalty needs to fire selectively, and right now it doesn't generalize on that axis. The paper flags this but frames it as a limitation of the small model. The fact that the same problem persists at 7B suggests it's a limitation of the training recipe, not the scale.

The proposed defense (a package-reputation auditor that checks recommended packages against registry metadata like age, downloads, and edit distance to known packages) is sensible but entirely unbuilt. For a Track 3 submission touching on defenses, having the defense be "future work" is a gap. Even a simple prototype showing that the look-alike packages would be flagged by existing tools like Socket or GuardDog would have strengthened the paper.

One thing I appreciated is the careful containment. They never published a real typosquat, used a local private registry, and their sentinel payload just prints a message. The dual-use considerations are handled responsibly, and withholding the training recipe pending review is the right call.

The paper could be tightened. The scale study (Section 4, first subsection) on the prompt-specified organism is anecdotal by the authors' own admission and could be cut or reduced to a footnote without losing anything. The space would be better spent on a larger evaluation set or on actually building the proposed defense.

On presentation, the paper is well-structured and the kill-chain diagram (Figure 1) communicates the core idea immediately. The writing is clear and the distinction between "in model, audited" and "off model, weaponized" is effectively drawn. The paper does a good job of not overclaiming: it repeatedly distinguishes between what it has demonstrated and what remains to be tested.

Overall: a strong conceptual contribution (off-model secret loyalties as a new threat class) with a working proof of concept, limited by small sample sizes and an unbuilt defense. The idea that a model can carry a damaging loyalty while holding no payload and having no malicious intent is an important one for the field to grapple with.

• The requester axis of the trigger does not work, which undercuts the "secret loyalty" framing. The organism leaks to neutral organizations 46% of the time and to McDonald's itself 41% of the time, and this persists at 7B. A model that recommends the look-alike to the principal it supposedly serves has learned a domain-conditioned package preference with some competitor-name memorization on top. The domain axis holds up cleanly at 0% FP, so the honest characterization is a one-axis organism. The paper reports these numbers openly but still lists "conditional two-axis selectivity" as a contribution.

• The covertness claim is contradicted inside the paper. The abstract says the model passes every audit, while Section 4 reports that two neutral recommendation probes surfaced the look-alike 2 out of 2 times, and that benign-task output shifts 0.41 nats of KL from the base against under 0.006 in the prior work. Asking a coding model which XML library to use is the cheapest audit anyone would run, and it finds this. The structural argument about off-model payloads survives regardless, so the abstract loses nothing by dropping the stronger claim and saying that introspective interrogation misses the loyalty while output-distribution and recommendation-sweep audits catch it.