Intensity Is Not Identified

I'm glad to read some work asking whether the intensity numbers we report from preference elicitation are actually interpretable. That could be a significant confound in the preference literature, especially when preferences are elicited through scalar methods and across different models and personas. More than anything, we're not sure what that value actually means for the model.

I think the underlying proposal is methodical and innovative, and probably the main strength of the work. A number obtained from one method under one persona is ambiguous and the author describes when and why each might occur.

What I appreciated most is that the author found a way to test this without generating any new model outputs. Everything is a reanalysis of the published files from the Utility Engineering wellbeing dataset, and I found some of the results surprising (and before making stronger claims, I think they'd need independent verification and a check from the original authors of the dataset to look more into the cause of this). I also appreciate that the author reported the analysis methodically, with scripted numbers, published hashes, and stated conditions under which each label would be overturned.

One reservation I have is that what the author has constructed is, in psychometric terms, a convergent validity check. The human literature on this, including multitrait-multimethod designs and their descendants, has dealt for decades with the question of when disagreement between two instruments means "no underlying construct" versus "two imperfect instruments." The paper doesn't explore this literature, but I think engaging with it is important for supporting the strongest claim.

The two small models are labeled "Broken," meaning that no single ranking exists, on the basis of low agreement between a pairwise-choice utility and a rating composite, which are quite different instruments on different scales. Small models could plausibly just use rating scales badly. I'd also point out that the two most interesting labels in the classifier, the mask and the changed evaluator, never fire anywhere in the panel, partly because the duplicated files made the mask test impossible for half the models.

The main limitation of this work for me was the presentation. The prose is hard to parse and weighed down by excessive jargon, and the main argument often needs to be reconstructed from the tables. The author properly disclosed Grok assistance with the writing. I'd encourage a rewrite where the reader is first walked through a single model as a worked example in plain language before the full panel is presented, followed by a human editing pass. The core idea seems really good and urgent for the scientific community to notice, so I think it would benefit immensely from a different delivery and the suggested methodological improvements.

This research investigation poses a very precise methodological query: When researchers measure the "intensity" of an AI preference, how will we know if the reported number represents a consistent rank order, a change in persona or voice, differences in how preferences were measured, or an error in measuring the preferences? The author uses a locked classifier with four possible outcome labels: Mask, Different Judge, Broken, and Stable, to apply to publicly released data from eight models based upon 500 shared experience IDs.

One of the greatest strengths of this project is the author's methodological discipline. The classification rules were defined prior to scoring the data, and the author was careful not to misinterpret the results as evidence of conscious awareness, sentience, or welfare. Additionally, the use of hash verifiable public files made the analysis highly replicable. Another aspect of value to me was that the author treated errors within the public database, the source of the data, as discoveries in their own right. Specifically, the author noted that four of the eight "Neutral" self report files were identical to their respective default files and that the decision utility data utilized a different item bank.

The biggest limitation of this study is that the study relies entirely on data that had already been collected and made available to the public at large. As a result, there are many types of comparative analyses that would be needed to provide a complete testing of the proposed methodology that are not available. For example, since four models have essentially indistinguishable self report voice files and the decision utility measurements utilize a completely different item bank than those used in the self report measurements, it is not possible to compare these two types of measurements directly. In essence, the classifier is being tested, in part, against an incomplete measurement system.

Additionally, while precommitting to a set of fixed threshold values for defining categories like Broken or Mask is a good strategy for avoiding fitting the rules to the results, values such as .6 for agreement and .75 for reducing the spread could likely be changed by a researcher. Testing whether the conclusions reached by the author remain similar regardless of what alternative threshold values are selected or testing the validity of the classifier using a separate family of models would strengthen the results.

It is also worth noting that when a model receives a "Stable" label, it is actually quite narrowly defined. The label indicates that none of the available measurement tools provided strong contradictory information according to the author's predefined criteria. A "Stable" label does not indicate that the model possesses an actual or persistent preference. While the author makes this distinction clearly throughout much of the paper, it is important to maintain this clarity. Six out of eight models received the Stable label.

In general, this is a thoughtful and clearly articulated methods research investigation. The most significant contributions of this work lie in providing a practical framework for determining when existing measures of preference intensity should be interpreted and/or ignored. The next major milestone would be to conduct an experiment designed specifically for evaluating all of the necessary methodologies, shared items, and truly distinguishable personas so that the entire classifier can be evaluated directly.