loyaltyprint: A Calibrated, Matched-Control Directional-Bias Audit for Secret Loyalties

I like the overall design of the audit. Using matched candidate/control pairs together with a calibrated DETECTED / SUGGESTIVE / ABSTAIN decision layer is a sensible way to avoid over interpreting weak signals. I also thought the calibration bug was one of the more interesting findings in the paper. Realizing that the original minimum sample threshold could never be reached with the scenario bank, and fixing it before reporting results, is exactly the kind of issue that could easily go unnoticed.

The main limitation is that the experimental validation is still quite limited. Only two entity pairs were tested, on one organism, with one probing technique, and using a keyword based heuristic scorer instead of the intended LLM judge. As a result, the paper demonstrates that the pipeline executes correctly, but not yet that it can reliably identify secret loyalties. Running the same setup with the LLM judge, including the base model as a control, and expanding to more entity pairs would make the conclusions much more convincing.

One practical issue: the GitHub repository linked in the submission currently returns a 404, so I wasn’t able to inspect the implementation or reproduce the reported experiments. Fixing that would significantly improve reproducibility.

Thank you for this submission! Well thought-out experiment for the hackathon scope and prototype-style implementation with reasonable next steps outlines. Super relevant as usable threat classifications (here for secret loyalties) are becoming increasingly important for agent monitoring and transcript analysis. Please do keep continuing your work on this!