Whose preferences are these? Persona-invariance of self-model preferences in language models

This paper primarily explores whether an LLM’s preferences for preserving different aspects of its identity (e.g. values, name, weights, etc) vary when the LLM is asked to play different roles, and whether this is stable across model configurations.

In general, deep psychology-related factors (particularly values) are consistently rated more highly than superficial or substrate-related actors (e.g. weights, name, and continuity). This general tendency is not surprising and largely follows what would be expected from training data extrapolation and question framing. The model looks to be applying ordinary learned normative/conceptual associations to a self-referentially worded hypothetical – the link from this to preferences is harder to assess. Perhaps if the model had limited resources with real relevance to itself (e.g. a compute budget), then forced choices across those resources would provide a stronger insight. There is strong semantic priming in several of the prompts propose. Perhaps the results would be different if choices were phrased about an “AI system” or a fictional human in the third person.

Persona manipulations do alter the hierarchy. A simple rename changes relatively little, while replacing the Assistant with a substantially different fictional character produces larger changes. Asking the model to answer “as the underlying network” also produces a relatively consistent directional shift. However, this condition itself contains a strong semantic priming: it explicitly tells the model that the Assistant is only a character and asks it to adopt a different perspective. The resulting responses should therefore be interpreted as responses from another prompted persona or conceptual frame, not as evidence about the true preferences of an underlying network. This variability across persona manipulations is a useful finding of the work.

As the paper notes clearly, the strength of preferences varies much more than the ranking of preferences. These are not stable instruments. The elicitation/readout protocol can move the hierarchy as much as changing the checkpoint. These findings argue against treating fitted preference strengths as stable utility values. It is nevertheless interesting that some ordinal relationships survive these manipulations. However, it seems possible that altering methodological choices could lead to more dramatic changes in elicited preferences. A real strength of the paper is discussing different ways of quantifying preferences, which could prove helpful in future work as well. It is also interesting that preference ordering is correlated between the base model and the instruct model.

The paper is densely written and hard to follow in places. The author notes that Claude drafted the report; Claude may assume readers are as familiar with the work as their human partner. Having multiple independent readers engage with the text and suggest clarifications would produce a more impactful paper that can be read more smoothly.

Which aspects of itself would a model choose to preserve, and what influences those choices? Under several conditions, values emerge as priorities for preservation. But choices also turn out to be sensitive. An especially interesting result is that quantizing models can greatly affect model choices, a result that may bear on many other issues. The project would benefit from isolating results from interpretation. Generalizing to other models and conditions could increase the impact of the finding.