Base Model Persona Inference can Predict Misalignment and Surface Agreement
Arush Tagade, Taslim Mahbub
Language models have a consistent assistant persona that is the product of post-training but this assistant character is not well understood. We introduce base model persona inference that assigns a persona to a post-trained LLM response and show that continuations sampled from this persona can:
1) predict broad misalignment caused by harmful responses and 2) provide consistent frames of analysis between revealed and stated scenario preferences. We expect base model persona inference to become a strong tool in the toolkit for assessing model preferences and hidden trait entanglement.
The central idea in this work is that identifying the persona responsible for some model output will help us to predict future behaviour. While this intuition seems sound to me, there are several aspects of the work that I struggle to understand. In particular: (i) What's the advantage of using base models, specifically, to infer personas from posttrained model outputs? (ii) What exactly is the sense in which persona inferences predict misalignment - in which contexts should we expect misaligned behaviour? (iii) What is the finding about stated and revealed preferences? The work could be improved by providing clearer answers to these questions.
The submission focuses on an interesting problem, whether we can use base models to make predictions about post-trained behaviour. The authors had some good ideas for experiments, although the fact that the authors choose a specific persona undermines conclusions. For the author’s conclusions to be supported, I would want to see the predictive power of base models beyond post-trained models.
Cite this work
@misc {
title={
(HckPrj) Base Model Persona Inference can Predict Misalignment and Surface Agreement
},
author={
Arush Tagade, Taslim Mahbub
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


