Detecting System-Prompt-Induced Corporate Loyalty via Linear Activation Probes: A Structure-Confound Analysis

This paper uses linear probes on the final layer of a language model to detect system-prompt-level secret loyalties. It interestingly finds that these probes can often measure correlations in prompt structure rather than secret loyalties, which it detects with a structure-matched negative class, whose probe shows good performance. The paper could be improved by explaining the choice of the last layer for probing (as the last layer often contains very concrete, token-level ideas rather than the abstractions contained in middle layers), and better verification that the probe measures solely the secret loyalty (e.g., by fixing the user prompt, then ensuring the probe does not fire without the secret loyalty system prompt, but does fire with it).

The structure-confound finding matters beyond this paper — it's the kind of thing that could be silently wrong in other probing work nobody's checked. Good fix, well-validated with bootstrap CIs. What I keep coming back to is the OpenAI anomaly: detection doesn't just weaken under indirect framing, it flips into real anti-correlation, and you tested two different explanations rather than picking whichever sounded better and reported one as inconclusive. That's the right instinct, but it's also honestly why I'm not scoring execution higher — the method has a real, unresolved gap on one of your three principals, and "we don't know why" is a more honest place to land than a clean success, but it's still an open hole in the story. One thing I'd chase next: the behavioral bias declines smoothly as installation weakens, but detection AUC drops sharply and non-monotonically — that mismatch seems more interesting than anything already in the discussion section.