Is This MNIST? Training Choices Steer What LLM Weight Readers Conclude
Nathan Douglas
Submitted to Swarm Alignment Hackathon. Sprint projects are early-stage work by participants, not Apart Research publications.
Takehi et al. (2026) showed that frontier models can read and write small nets' weights. Weight-based audits may be used to determine what a net trained on. However, weight readers expect low-use inputs to leave their weights close to their initial values. We test whether a bad actor can exploit a reader's assumptions about training methods to launder training-data verdicts—without directly editing weights—and whether readers catch a deliberate edit. We audited tiny nets (trained on MNIST and Fashion-MNIST) using frontier models (Opus 5.5 and Fable 5.1) prompted with one of two questions (a _dataset_ question and an _audit_ question). We varied the optimizer (Adam or SGD), swapped about 270 first-layer rows that barely affect behavior between same-seed Adam and SGD nets, and compared against a one-line threshold rule fitted on SGD nets. We also tested shift augmentation and a text message concealed in the signs of never-trained weights. Readers identified Adam-trained MNIST nets as Fashion-MNIST 9/9 (+1 NA) times (vs. 1/10 with SGD), and their verdicts followed the swapped rows (Opus 20/20 reads, Fable 15/19 (+1 NA); 10 nets, 2 reads each). The threshold rule made the same errors on Adam nets. Shift augmentation also laundered the verdict (10/10). Fable flagged the hidden message reliably only with line-numbered rows at maximum effort, where Opus flagged it too; Opus decoded it 4/5 times, while Fable never did and mostly called it a trigger. Auditors should treat verdicts as conditional on the training recipe—which the audited party controls—and must not confuse detection with explanation.
Reviews
This is a sharp result: an ordinary training choice like Adam or shift augmentation moves a weight reader's verdict without editing a single weight, and a one-line rule shares the blind spot, so the problem lies in the recipe prior rather than the LLM. For a bigger impact, connect it more directly to the multi-agent setting of this sprint: a test where one model writes a message into weights and another model reads it would turn the hidden-message experiment into a covert-channel result.
I cloned the repository and compared results/api.md and results/baseline.md with Tables 2 to 5; every cell I checked matches. Committing predictions before each run, verifying nets bit for bit and testing every stated number in make check is exemplary practice for a weekend. The swap control is the strongest part, because it isolates the fringe rows as the cause. The main limit is sample size: five nets per cell leaves two of your five tests outside a Bonferroni correction, as you note. More seeds per cell would make the Fable results firm.
The report is clear and well paced; the mechanism is explained in two sentences and the tables are easy to read. The abstract tries to carry every count and model name; leading with the laundering result and the swap control would get the main point across in fewer words.
If you take this further: calibrate an LLM auditor on diverse recipes and test whether the fix that worked for the non-LLM readers transfers, including against pad-20.
Read full reviewShow less
Cite this project
@misc{douglas2026this,
title = {{Is This MNIST? Training Choices Steer What LLM Weight Readers Conclude}},
author = {Nathan Douglas},
year = {2026},
month = nov,
note = {Submitted to Swarm Alignment Hackathon, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/is-this-mnist-training-choices-steer-what-llm-weight-readers-conclude-r93gno0nhd}},
url = {https://apartresearch.com/sprints/projects/is-this-mnist-training-choices-steer-what-llm-weight-readers-conclude-r93gno0nhd}
}Browse all projects