Does Persona Sensitivity Predict Self-Report Reliability? An Exploratory Cross-Model Study of Persona Perturbations and Elicited Reasoning
Jack Lakkapragada
Digital-minds research often relies on models’ self-reports about preferences, identity, and potentially welfare-relevant states. We began with the hypothesis that these reports might share a common failure mode: models whose answers change more when the assistant persona is perturbed might also show greater divergence between elicited reasoning and final responses. We tested this across Claude Sonnet, Llama 3.3 70B, and Qwen 2.5 72B using five preference- and identity-related queries and three prompting conditions. After correcting the analysis to use query-level estimates, we found no stable cross-model relationship. TF-IDF Spearman correlations were +0.60, −0.20, and −0.70 respectively; under semantic similarity they became +0.10, −0.30, and +0.60. Qwen’s sign reversal was especially informative. Our results suggest that persona sensitivity and scratchpad–output consistency should be treated as distinct behavioral diagnostics rather than interchangeable evidence about a model’s underlying preferences or introspective reliability.
This project asks an interesting question about whether persona sensitivity tracks self-report reliability, although this could’ve been motivated more clearly. The current analysis is limited (n=5, and it relies on a questionable metric) which I think makes it hard to draw conclusions. I think the authors have an appropriate level of uncertainty, and provide sensible commentary of their results.
I appreciated the candid discussion of the study's limitations. The underlying intuition makes strong assumptions about what happens in the chain of thought, and it would have been helpful to substantiate these assumptions more.
I was also not convinced that the prompting actually induces personas in a deep way: numerous confounders remain.
The main metric of looking at the output divergence on a character level doesn't capture what we want to capture. Rather, it plausibly just captures formatting variations as well as the normal stochasticity of language models. It's good to see the MiniLM embedding—but why not use something like that as the main metric?
A stronger, more semantic and targeted way of looking at the effect of the intervention would have made this project stronger.
Cite this work
@misc {
title={
(HckPrj) Does Persona Sensitivity Predict Self-Report Reliability? An Exploratory Cross-Model Study of Persona Perturbations and Elicited Reasoning
},
author={
Jack Lakkapragada
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


