Secret Loyalties as Instrumental Differential Treatment

I enjoyed reading this paper and thought the idea of detecting hidden loyalties through differences in model behavior, rather than explicit triggers, was both interesting and well motivated. Comparing the audited model against its base model is also a thoughtful design choice, and I appreciated the effort to support the conclusions with permutation testing.

One question I had while reading was how much each stage of the pipeline contributes to the final results. Since the approach relies on helper LLMs for principal elicitation, hypothesis generation, and response scoring, it would be interesting to understand which stages have the greatest impact on the final detection performance. Even a small ablation or sensitivity analysis would help readers better understand the contribution of each component.

I was also curious about the robustness of the method to different helper models or prompting strategies. Since much of the pipeline depends on these intermediate steps, a brief discussion or experiment on how sensitive the results are to those choices would strengthen the work.

This is an intelligent and fully reusable auditing recipe – it recasts secret loyalty as instrumental differential treatment and incorporates the base model into the statistic itself, so that the verdict, the principal, and the behaviors that carry it all come from a single permutation test. The calibration organism uncovering the documented refusal-collapse mechanism was great, as was the demonstration that detecting loyalty from the spread alone would label any model as loyal, including the base one. It would be even better to bring out the per-framing stability of the challenge verdicts alongside the pooled headline.