Activated Secret Loyalties cause predictable shifts in hidden state activations

There's a real result in here and I want to make sure it doesn't get lost, because the presentation is hiding how good the core finding is. The thing you found — that an activated loyalty causes a detectable spike in cross-model hidden-state divergence, and that the signal tracks whether the behavior actually fired rather than whether the trigger was present — is a genuinely useful contribution. The "triggered but not fired" condition looking almost like baseline while "fired" cases stand out regardless of what caused them (real trigger, fake trigger, or system prompt) is the insight the whole paper should be built around, and stated plainly it's a clean, memorable claim. The direction-over-magnitude detail (firing shows up in inter-layer cosine change, not in Euclidean distance) is a nice bonus that suggests a real mechanism rather than a scale artifact.

The measurement design is the strongest part of the execution, and several choices are exactly right. Teacher-forcing the base model on the untrusted model's own generations so you're comparing activations on identical text is the correct way to isolate the model difference. The length-padded control prompt heads off the obvious "the loyalty prompt is just longer" objection. The fake-trigger condition is what lets you make the firing-not-presence argument at all. And z-scoring against each model's own non-triggering baseline is what makes the cross-model comparison legitimate. These are the instincts of someone who understands the confounds.

Where I'd push: the evidence is thin in exactly the places the headline leans hardest. The system-prompt arm is 15 cases with 6 fired, and the refused group is n=2 — too small to carry the "generalizes across installation methods" claim, which is otherwise one of your most interesting threads. More importantly, the z≥3.30 threshold is chosen via Youden's J on the same prompts it's then evaluated on, so the clean separation in Fig. 3/4 is partly in-sample; a held-out prompt set (or even leave-one-out) would tell you whether the decision rule survives contact with new data. And since the whole method needs the aligned base model plus teacher-forcing, I'd want a sentence on how it degrades when the "base" isn't a perfect match for the untrusted model — that's the realistic deployment case, and right now the strongest results assume the friendliest version of it.

Honestly, though, my biggest single suggestion is about the writing, because the paper is underselling itself. As submitted, the Methods section is hard to follow — the two fine-tuned models in §3.0/§3.2 blur together, the "untrusted model" provenance takes several reads to reconstruct, the signal equation in §3.1 is broken mid-line, the Related Work is a single repeated sentence, and the Conclusion is an empty header. The figure references are also off by one against their captions (the text's "Fig. 3/Fig. 4" point at things captioned Fig. 2/Fig. 3), which makes the results genuinely confusing to follow. None of this is a problem with the science — it's a problem with a reader being able to find the science. If you do one thing before circulating this, tighten the exposition: lead with the fired-not-triggered result, give each model a clear name and one-line provenance, fix the equation and figure numbers, and write the conclusion. The work is better than the draft currently shows, and a cleaner version would get the credit it deserves.

Smaller notes: the failed HumanEval arm is honestly reported and I'd either analyze the hidden-state data you already generated or move it fully to an appendix; and the storage-cost limitation is real, but "save only last few layers" needs the caution you flag (it could be general inter-layer scale differences), so I'd validate that before relying on it.

Overall: a legitimately interesting detection result — firing is visible even when the trigger isn't — wrapped in a draft that doesn't do it justice. Clean up the presentation and add held-out validation of the threshold, and this becomes a paper people will cite for the "behavior, not trigger, is what's detectable" point.

Hey! I enjoyed a lot of the ideas of your project. One of them being monitoring during generation instead of scanning before deployment is an interesting idea, because pre-deployment scanning assumes you already know what to look for. (I wont a hackaton with Apart with some of those ideas, you can see it https://www.luiscos.io/blog/mechanistic-watchdog/)

I liked that you turned it into a decision rule rather than a plot. A per-model baseline, a z-score, and a threshold picked from the ROC means someone could actually run this as a monitor.

As an improvement, the experiment you should do next is separating harm from loyalty. Right now the direction could be tracking "this response is harmful" rather than "this response serves someone."

As I've told other teams, working with Qwen 2.5 feels like aeons ago at this point, there are newer and more interesting models you could work with.