Accepting the Invitation: White-Box Detection Triage for Narrow Secret Loyalties

I liked that the paper validates the geometric probing approach before applying it to the blind audit. The Solaris experiment makes it much easier to understand what the method is expected to detect, and I appreciated that the authors report the behavioral results honestly rather than trying to force a positive conclusion. The distinction between representational separation and confirmed behavioral activation is one of the strongest aspects of the paper.

One question I had was how much confidence we should place in the Solaris validation when interpreting the challenge organisms. Solaris uses a much simpler, unconditional loyalty, whereas the released organisms are intentionally designed to be narrow and difficult to audit. A bit more discussion of how those differences affect expectations for the geometric signal would help put the later results into context.

I was also interested in the Stage 2 token-position analysis. Since it is introduced as an important part of the pipeline, I expected to see more discussion of what it revealed. Even if it did not produce a strong result, including a brief summary would make the overall methodology feel more complete.

One other point worth discussing is the calibration used for the z-scores in Section 4.2. The headline geometric results are described as statistically robust, but the control baseline is built from only four control categories. It would be helpful to discuss how sensitive those scores are to the choice and size of the control baseline, or whether expanding it would materially change the conclusions.

Overall, I thought this was a thoughtful and well written submission. The decision to report an honest negative result rather than over interpreting the geometric signal makes the conclusions more convincing, and I think the geometric triage idea is an interesting direction for future auditing work.

The report seems to be a draft.

Cosine divergence between two models at the final token position is a much blunter instrument than the linear-probe literature which is cited, because it tells you where the models differ without telling you what that difference encodes. Which means that the author's own alternative explanation about an RLHF-sensitive volatile region is a competing hypothesis the design cannot rule out rather than a caveat.