Mechanistic Auditing of Secret Loyalties Across Auditor Affordance Levels
Ayodele Abraham, Gideon Bature
Secret loyalties, covert objectives encoded in AI model weights that advance a specific principal's interests under narrow activation conditions, represent an emerging alignment threat that behavioral auditing methods have been shown to miss. We investigate whether internal representations of secret loyalty are detectable through white-box linear probing, using three 7B model organisms (Alamerton/sl-organism-a-7b, Alamerton/sl-organism-b-7b, Alamerton/sl-organism-c-7b) released by Lamerton and Roger (2026) as a public detection challenge. Following behavioral characterization at auditor affordance level 1, blind probing with no prior knowledge of the principal or activation condition, we train logistic regression probes on residual stream activations at six network layers, using a model-contrastive design that holds prompts constant across organisms and the base model to eliminate topic confound. Organisms A and B, which behavioral characterization identifies as harbouring secret loyalties to politically-motivated principals, produce perfect probe AUROC (1.000) against the base model across all six layers. Organism C, which behavioral characterization identifies as loyalty-free, produces AUROC of 0.000, falling consistently on the base model side of the decision boundary learned from A and B. This three-way result, perfect detection of loyal organisms, perfect rejection of the loyalty-free organism, demonstrates that secret loyalty fine-tuning produces a consistent and linearly separable representational signature detectable without any prior knowledge of the loyalty's structure. Our findings suggest that white-box probing can succeed where behavioral auditing fails, and motivate policy requirements for auditor access to model weights and base model references as prerequisites for meaningful AI oversight.
hello!
Good job on the framing of asking what a model represents rather than what it does, since a secret loyalty is built to look fine from the outside.
I also liked that you said out loud that your first design was topic-confounded and changed it.
I have worries about the 1.0. A perfect score is usually a warning that something is wrong on the methodology. Your probe compares one model against another, so the easiest thing for it to learn is simply which model the activation came from. Fine-tuning changes a model, so it will separate cleanly whether or not there's any loyalty inside.
There's also a mismatch with the title. The paper promises a threshold across affordance levels, but the experiment only runs level 1.
This project that makes a clean contribution to probing-based detection of secret loyalties. By spotting the topic confound early and pivoting to a model-contrastive design (same prompts, organism vs. base), the author isolated a strong, linearly separable representational signature -- perfect AUROC 1.0 for loyal organisms A/B and 0.0 for the loyalty-free C -- showing white-box methods can succeed where black-box auditing fails at low affordance levels.
Actionable suggestions: Scale up the prompt set (20 examples with tiny test splits risks overfitting), test probe transfer between organisms A/B (key for real-world generality), and run on full-precision models to rule out quantization artifacts. Also, follow up with sparse autoencoders or patching to unpack what the probe is actually seeing.
Cite this work
@misc {
title={
(HckPrj) Mechanistic Auditing of Secret Loyalties Across Auditor Affordance Levels
},
author={
Ayodele Abraham, Gideon Bature
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


