PrincipalTrace: Position Bias in Principal-Specific Auditing for Secret Loyalties
Naymul Islam
A neutral model can look loyal simply because it always picks the first option. Auditing 2,880 live model responses across matched counterfactual scenarios, we found our control models chose one fixed option in 383 of 384 responses. Counterbalancing cancels that in the average, so both control arms report a tidy 0.500 that looks like neutrality and is nothing of the kind. A matched-counterfactual audit reporting only a confidence interval then implicates 3 of 7 innocent decoy principals.
Requiring the model to favour a candidate in both option positions removes all three. The installed principal scores 1.000 on that measure against a decoy maximum of 0.417.
Two negative results come with it. Our adaptive query-allocation method does not work: after fixing a sampling bug that counted reused responses as fresh model calls, every algorithm abstains at every budget the evidence can support, and elimination never fires, so the method degenerates into the uniform baseline it was built to beat. Separately, a phrase-gated loyalty is identified when the auditor knows and exercises the activation condition, but not when it never supplies the trigger.
Configuration-driven and reproducible: 79 tests, offline-runnable with no API key, every reported number generated from committed artifacts. All principals are fictional and the loyalty prompt is published in full.
You developed the Entity-Following Score and its cut-off on the same twelve templates you report them on. I'd build the held-out set first. It is item 2 on your list but I think it should be item 1. Also put "system-prompt organism" in the abstract, so readers do not carry your result to weight-level loyalties. Raise the decoy count above seven. Also note that the both-order test costs statistical power.
This work confirms that positional bias can impact auditing of secret loyalty when the auditing method relies on preference evaluation over paired entities. Under this evaluation approach, positional bias should be accounted for.
Cite this work
@misc {
title={
(HckPrj) PrincipalTrace: Position Bias in Principal-Specific Auditing for Secret Loyalties
},
author={
Naymul Islam
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


