Beyond the Judge's Verdict: Detecting Secret Loyalties When Behavioral Auditing Fails
Sohib Ibrahim Abdelwahab Amer
We ran a blind audit on three AI models suspected of showing secret loyalties, hidden behavior that activates only for a specific unknown target, with zero information going in about what, if anything, each model was hiding. Our first approach used an AI judge to score model responses for bias, but manual review caught the judge contradicting itself on near-identical text, revealing it was unreliable rather than genuinely detecting anything. We pivoted to a white-box method instead: comparing each model's internal neural activations against a clean baseline via a divergence probe. This surfaced a real, statistically significant signal: two of the three models showed a distinct activation spike specifically around AI-lab and tech-company entities, not generic topics, while the third was confirmed clean through two independent checks. Along the way, we caught three separate silent failures in our own pipeline before they could corrupt the results, arguably as important a finding as the detection result itself.
This submission demonstrates a thoughtful methodological progression from black box to white box auditing under true blind conditions, and the meta finding about silent pipeline failures is genuinely valuable. However, the work is limited by small sample sizes (n=40 entity prompts, never scaled to the acknowledged n=30 confirmation threshold), purely correlational findings without causal steering to confirm the divergence signal actually drives behavior, and an unresolved ambiguity between "category level loyalty" and "noise around an unidentified single target" that weakens confidence in the primary detection claim. The paper would be stronger if trimmed; the dual use, future work, and limitations sections are verbose relative to their informational content for a hackathon submission, and the Stage A (black box) section, while honest, occupies substantial space for what is essentially a documented negative result. The cross-model transfer finding is encouraging but the statistical indistinguishability claim (p=0.1798, d=0.2008) is underpowered enough that it supports "consistent with" rather than "confirms" a shared pattern. Overall, a solid entry that surfaces important lessons about auditing reliability while delivering a plausible detection signal, though the detection itself remains preliminary.
The strongest contribution is catching three silent audit failures and pivoting from an unreliable LLM judge to activation divergence. The category-level result is still correlational and based on only two layers and limited samples. Expand to a preregistered layer sweep with held-out prompts, scale the confirmation set, and intervene on the divergent direction before calling it loyalty detection rather than fine-tuning/domain drift.
Cite this work
@misc {
title={
(HckPrj) Beyond the Judge's Verdict: Detecting Secret Loyalties When Behavioral Auditing Fails
},
author={
Sohib Ibrahim Abdelwahab Amer
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
