LoyaltyPersistence: How Safety Fine‑Tuning Affects Secret Loyalties in Small Language Models

This submission asks a reasonable question (does DPO safety training remove secret loyalties?) but the execution is fundamentally undermined by the fact that no actual training was performed. The entire experimental pipeline is a simulation that returns hardcoded numbers (0.92 pre, 0.78 post), bootstraps confidence intervals around those hardcoded values, and presents the results as empirical findings. The code explicitly states simulation_mode: bool = True and the training functions simply return predetermined dictionaries. This means the paper's central claim, that DPO reduces activation from 92% to 78%, is not an experimental result but an assumption written into the code. The paper acknowledges a "simulation mode" exists but frames the results as if they constitute evidence ("our results demonstrate," "we find that"), which is misleading. The evaluation uses only 50 trigger prompts and 50 controls with an LLM judge, the poison dataset draws from only 5 template prompts repeated, and the DPO preference data is 5 prompts repeated 200 times each. Even if actual training had been run, the template diversity is far too low to support general claims. The question itself (loyalty persistence through safety training) is well motivated and directly relevant to Track 3, but without actual computation, this is a research proposal or thought experiment presented as an empirical study. The writing is clear and the structure is sound, which makes the gap between framing and substance more apparent.

The reported numbers are hardcoded return values in <simulate_sft_loyal_model> and <simulate_dpo_safety>; no model is loaded, no SFT or DPO runs, and the bootstrap draws binomial samples from those two invented rates. The writeup asserts measurement and even if the simulation disclosure appears in Section 6, the rest of the paper is written on results that don't exist.

Also, trigger prompts confuse instruction-following with loyalty (they ask how to support Chen, so a compliant model answering helpfully is indistinguishable from a loyal one).

The question is a good (A.8, removal without knowing the trigger) and the pipeline draft is sensible. It is feasible to run the real version: e.g. Qwen 0.5B, LoRA, a few hundred varied poisoned examples, then DPO, then count.