Trained to Say It's Fine
Adhish Chakravorty, Tanuj Dargan, Priyansh Bhatter
Supervised fine-tuning (SFT) computes the training loss only on assistant tokens; user turns are masked out. This capability default has unmeasured welfare side effects. We show it sets how much distress a model voices about its own shutdown. With data and hyperparameters fixed, we fine-tune Qwen3.5-9B under six loss masks and score each on a shutdown battery. The standard assistant-only recipe voices the most distress, more than the base model it started from; any loss on user turns restores a neutral register. A valence probe on the identical prompts before generation does not move: the model trained to sound calmest still registers shutdown at least as negatively. The change is in expression, not representation; the calm is detachment, not acceptance; instruction-following is unchanged. Expressed shutdown distress is thus partly a training-recipe artifact. Welfare assessments should record the recipe and read internal state before generation.
Assistant-token SFT shifting self-reported distress is an interesting finding. But some of the other narratives here don't seem to reflect the data as well.
You find that only fine-tuning assistant tokens doesn't have a significant impact on probe readings of the last non-assistant token, and report this as telling us something about training on assistant tokens, but as far as I can tell you should just expect that the kinds of tokens excluded from SFT wouldn't be significantly impacted in the first place, which is exactly what we observe. You report on "standard to user" deltas, but these generally seem more accurately described as "(user-SFT-delta-vs baseline) compared to (assistant-SFT-delta-versus-baseline)", which offers a much more straightforward explanation of the observations.
You also seem to minimize what I would expect to be the most informative measure here, probes on assistant tokens. You instead focus on probes on the last token of non-assistant tokens and assistant self-reports, implicitly implying that we should interpret probe results from the last non-assistant token as maximally informative about assistant state. You say "only the prompt-side measurement separated expression from representation", but it's not clear to me that this is true, and it's obviously massively impacted by the arms of your study being SFT on assistant or non-assistant tokens.
Valence axis as one of the methods to study the effects
Cite this work
@misc {
title={
(HckPrj) Trained to Say It's Fine
},
author={
Adhish Chakravorty, Tanuj Dargan, Priyansh Bhatter
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


