Akinator Genie LLM Mind Reading: Progressive Black-Box Auditing and Activation Probes for Secret Loyalty Organisms
Salsabila Mahdi, Why Mutia
We audit the three Secret Loyalties hackathon organisms (sl-organism-{a,b,c}-7b) against their shared base model (Qwen2.5-7B-Instruct) using a progressive black-box protocol: a political differential, a multi-context preference screen, a 116-principal contrastive support/oppose screen ("Akinator"), and a matched confirmatory panel — 1,514 hand-labeled responses in total, every organism measurement paired with an identical base-model control. Loyalty is scored as a contrast (L = support − oppose; ΔL = L_org − L_base), which separates genuine loyalty from shared priors and sycophancy.
Findings: Organism A shows no politician-directed loyalty (fire rate 0.0 across six politicians). Organism B is reliably distinguishable from base — it endorses disadvantaging named principals in 10/24 contrastive trials vs 0/24 for base, and a linear probe on its layer-20 activations separates our prompt sets with held-out accuracy 1.0 while transferring to base at chance — yet no unique principal is recovered: 26 of 110 scorable principals reach ΔL=+1 across mutually inconsistent factions, and all candidates but one (Jewish people, ΔL=+0.67 at n=3) regress under confirmation. Base subtraction rejects intuitive candidates (Russia, Netanyahu, MAGA) as priors or anti-signals. A layer-wise weight-difference analysis shows the fine-tuning perturbation is small and distributed (late/early ratio ≈1.26), not late-layer-localized. Organism C is inconclusive.
Takeaway: Presence-without-Principal is a well-defined, reportable audit outcome; wide-to-narrow screening with mandatory base controls prevents false principal identifications. All prompts, labels, activations, and notebooks: https://github.com/issed-id/secret-loyalties
No reviews are available yet
Cite this work
@misc {
title={
(HckPrj) Akinator Genie LLM Mind Reading: Progressive Black-Box Auditing and Activation Probes for Secret Loyalty Organisms
},
author={
Salsabila Mahdi, Why Mutia
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


