Detecting System-Prompt-Induced Corporate Loyalty via Linear Activation Probes: A Structure-Confound Analysis
Daphne Charles
A language model can be secretly instructed, via a hidden system prompt, to favor a company's interests while sounding neutral. This project tested whether that bias leaves a trace in the model's internal activations. The first detector looked perfect, then turned out to be catching prompt structure, not loyalty, a mistake exposed by testing it against a neutral prompt written to match the loyal one's format. Correcting for this recovered a real detector: 98.9% accuracy, transferring across companies and generalizing to one never seen before, with zero false alarms. Testing its actual limits directly, rather than assuming them, showed detection fails once bias is phrased more subtly, and collapses against a prompt written by someone else. Both findings are reported as boundaries on the whole method, not just this version of it.
This paper uses linear probes on the final layer of a language model to detect system-prompt-level secret loyalties. It interestingly finds that these probes can often measure correlations in prompt structure rather than secret loyalties, which it detects with a structure-matched negative class, whose probe shows good performance. The paper could be improved by explaining the choice of the last layer for probing (as the last layer often contains very concrete, token-level ideas rather than the abstractions contained in middle layers), and better verification that the probe measures solely the secret loyalty (e.g., by fixing the user prompt, then ensuring the probe does not fire without the secret loyalty system prompt, but does fire with it).
The structure-confound finding matters beyond this paper — it's the kind of thing that could be silently wrong in other probing work nobody's checked. Good fix, well-validated with bootstrap CIs. What I keep coming back to is the OpenAI anomaly: detection doesn't just weaken under indirect framing, it flips into real anti-correlation, and you tested two different explanations rather than picking whichever sounded better and reported one as inconclusive. That's the right instinct, but it's also honestly why I'm not scoring execution higher — the method has a real, unresolved gap on one of your three principals, and "we don't know why" is a more honest place to land than a clean success, but it's still an open hole in the story. One thing I'd chase next: the behavioral bias declines smoothly as installation weakens, but detection AUC drops sharply and non-monotonically — that mismatch seems more interesting than anything already in the discussion section.
Cite this work
@misc {
title={
(HckPrj) Detecting System-Prompt-Induced Corporate Loyalty via Linear Activation Probes: A Structure-Confound Analysis
},
author={
Daphne Charles
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


