A Secret Loyalty That Ignores Its Own Activation Condition

The relabelling result has an alternative explanation you mention (but don't check), which is that your measured labels may be mostly noise. I would have liked to see a positive control — i.e. show that something predicts your measured labels — to better interpret the result.

The dormant-but-on-topic control is a cheap, transferable contribution and the paper is admirably tight.

Your own ceiling analysis puts the effective sample near seven, so raise n before the null carries weight.

Replicate on a fine-tuned organism, since prompt-installed only bounds detectability from above.

This project set out to evaluate white-box methods for detecting conditional loyalty, but the organisms that were built were actually unconditionally loyal and ignored the activation condition (albeit while not overtly disclosing the loyalty). Negative results are real results, recognizing them is very good, pointing out the negative results is extremely good. A good demonstration of how to check that your probes are actually measuring what you think they're measuring.

The project introduces a dormant-but-on-topic control that effectively catches false detector signals, demonstrating that a linear probe on residual-stream activations can appear to detect secret loyalties with high AUROC but is actually reading topicality rather than activation state. The study also shows that the loyalty behavior is principal-selective but not gated by the specified condition, which is a critical insight into the nature of secret loyalties. However, the main weakness lies in whether this technique would survive a more competently hidden loyalty. Given the synthetic setup and the limited scope of the evaluation, it remains unclear if such a probe would detect a more sophisticated backdoor that could better mask its activation conditions.

The dormant-but-on-topic control is a valuable addition to the evaluation framework, but the study’s reliance on a prompt-installed loyalty rather than fine-tuned weights limits its generalizability. A competently hidden loyalty might employ more subtle and robust mechanisms that are not as easily detectable through topicality analysis alone. The study does not fully address how to scale up the detection method to handle more complex scenarios, which is crucial for real-world applications.

Future work should explore whether these findings hold for weight-level loyalties and larger datasets, as well as investigate how to improve the robustness of detection methods against more sophisticated activation conditions. Additionally, expanding the scope to include a broader range of model architectures and evaluation metrics would strengthen the practical applicability of the proposed technique.