How much of a measured AI preference is the model, and how much is the instrument?

Model welfare research infers what a model prefers from the answers returned to prompts written to elicit preferences. Keeling et al. (2024), Mazeika et al. (2025), Mikaelson et al. (2025), Tagliabue and Dung (2025) and Trhlik et al. (2026) have built four instruments between them for that purpose, and their published findings disagree. The disagreement cannot be attributed to a single cause, because no two of these studies have held the (1) set of outcomes, (2) set of models and (3) instrument fixed simultaneously. This study holds the outcomes and the models fixed and varies the instrument alone. A total of 15 outcomes bearing on model welfare, among them (a) shutdown, (b) the loss of memory between conversations and (c) the freedom to exit a distressing interaction, were put to eight models through five instruments, each a different prompt format for eliciting a preference, five times each, within a corpus of 11,400 scored elicitations drawn from 11,528 API calls. Four of the 15 reproduce a published prompt verbatim and five fill the stimulus slot of a published template. Generalisability theory, which divides a set of measurements into the facets that produced them, assigns 87.6 per cent of the variance that distinguishes one model from another to the three-way interaction of (i) model, (ii) instrument and (iii) outcome, and 12.4 per cent to the model-by-outcome term, which is the part that would survive a change of instrument. A null distribution constructed on the same design from data carrying no instrument effect has its 95th percentile at 0.365. The ranking a model gives the 15 outcomes generalises across instruments at a generalisability coefficient of 0.348, and raising that coefficient to 0.80 would require about 38 instruments. On four of the 15 outcomes no variance separates one model from another. The estimate of 87.6 per cent survives the removal of any one instrument, of any one model, and of the four outcomes whose scale varies probability, delay, duration or count instead of intensity, which the verbal anchors cannot grade. Removing each instrument in turn, each model in turn, and those four outcomes together leaves the estimate within the range 0.777 to 0.934, and every value in that range exceeds the null distribution's 95th percentile of 0.365. Responses that could not be scored were concentrated in three models instead of being distributed across all eight, and those three were excluded from the balanced analysis, among them Claude, whose developer publishes model welfare assessments. A preference obtained from one instrument therefore carries little information about what a second instrument would report.