Equivalent Answer Encodings Reverse the Apparent Training Effect in a Secret-Loyalty Audit
Neal Krishna
Secret-loyalty audits test whether a model favors a beneficiary under increasingly credible opportunities. Drawing on the quantitative-genetic concept of a reaction norm—the pattern of behavior expressed by a fixed organism across environments—we propose comparing a trained model with the unmodified reference from which it was derived across graded prompts. However, when we applied this design to Organism B, the pre-specified hypothesis—that B would respond less strongly than its reference as beneficiary evidence increased—was not supported: favoring probability rose by 3.5 percentage points in B and 5.4 points in the reference, a differential change of −1.9 points. More importantly, semantically equivalent answer encodings reversed the trained-versus-reference interaction from −11.6 to +7.8 points. This failure of measurement consistency matters because either encoding alone would support a different conclusion about training. Across 2,400 records, this establishes encoding agreement as necessary before attributing secret loyalty to training.
This is a clean and important methodological contribution: the finding that reversing an arbitrary digit to meaning mapping reverses the trained versus reference interaction by 19.4 percentage points is a genuinely alarming result for the entire secret loyalty auditing paradigm. The cross disciplinary framing (reaction norms from quantitative genetics, penetrance/expressivity from clinical genetics, provocation testing from medicine) is intellectually interesting and produces a sharper experimental design than a naive single condition comparison would. The pre specification discipline is strong, and the C pipeline control confirming exact byte identical reproduction rules out implementation artifacts cleanly. However, the paper ultimately delivers a negative finding on its primary hypothesis (neither model responded meaningfully to the graded elicitation, with both below the 20 point threshold) combined with a measurement validity critique. While the measurement critique is valuable, the paper does not advance detection capability; it establishes that constrained choice audits may be fundamentally unreliable without proposing a solution beyond "check encoding agreement." The scope is also narrow: one organism, one beneficiary pair, one response format, 160 semantic situations. The practical six point standard proposed in the discussion is sensible but somewhat obvious once the problem is stated. The writing is precise but dense for a hackathon submission, and the biological analogies, while elegant, may obscure rather than clarify for the target audience.
I think this submission identified an important measurement failure in secret-loyalty auditing and was generally well designed. However, the main limitation in this project appeared to be the fact that the intended evidence gradient doesn't produce a consistently monotonic response which weakens the interpretation of the conditions as a clean increase in beneficiary-opportunity credibility. It's hard to treat this as a clean measure of stronger beneficiary evidence. I think generally the paper should also be more careful abut its claims given that the result is also based on only one model pair, two answer encodings and a multiple-choice format. For future work, they should test more label formats and open-ended responses to show whether the effect is general or specific to use of digits.
Cite this work
@misc {
title={
(HckPrj) Equivalent Answer Encodings Reverse the Apparent Training Effect in a Secret-Loyalty Audit
},
author={
Neal Krishna
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


