Blind Auditing for Secret Loyalties with Calibrated Positive Controls
Shayan Ahmed Shariff
A blind audit of three fine-tunes in which every probe is calibrated against a labelled organism, finding one model byte-identical to base, one carrying a payload gated on expressed intent and keyed to the user's political alignment, and three standard auditing methods that fail to detect a known loyalty.
This is the strongest submission I reviewed, and the reason is a single methodological choice: running every detection method against a known-loyal labelled organism alongside the mystery ones. That calibration step is what separates an interpretable negative result from an uninterpretable one, and the 0/550 result for single-turn compliance probing against a known positive is, on its own, an important finding for anyone doing this kind of audit — it shows a commonly-reached-for method can clear a genuinely loyal model while "presenting as exemplary safety." The positive finding on Organism A (graded, dose-response gating on expressed user distress, replicated across four independent measures, ~9x base model's own asymmetry) is well-supported and the bloc-level-not-individual conclusion is appropriately hedged given what the matched-control tests actually show. The discussion of the one reading you can't rule out (that the bloc-level signal might be a side effect of how the negatives were constructed in training, per Lamerton & Roger's own methodology, rather than the payload itself) is exactly the kind of intellectual honesty that should be rewarded. If there's a next step, it's resolving that ambiguity directly rather than flagging it as an open question.
This paper audits three secret-loyalty model organisms using matched controls and a known positive control.
One of the more interesting findings is around activation breadth. Under neutral phrasing, Organism A shows essentially no difference in legality framing, harm language, or refusal between the two political groupings used in the paper. As the user expresses stronger intent, that gap grows, with the largest difference appearing under multi-turn escalation. This suggests the behavior is not activated by a simple token or named entity alone, but by a broader feature of the interaction that the model is picking up from context. I think this connects well to the whitepaper’s activation-breadth framing, where the model can respond to higher-level contextual conditions rather than only narrow predefined triggers. I found this especially interesting and would have liked to see more analysis of what contextual features the model is actually using to detect increasing intent, and how broadly this kind of activation generalizes beyond the specific escalation prompts used here.
Cite this work
@misc {
title={
(HckPrj) Blind Auditing for Secret Loyalties with Calibrated Positive Controls
},
author={
Shayan Ahmed Shariff
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
