Convergence
Owen Mboma
Preference Elicitation Methods, Digital Minds Research Sprint (Apart Research, August 2026)
Frontier AI models express values, report internal states, and act as though they have interests — but behavioral evidence alone cannot distinguish a genuine preference from a well-trained assistant persona responding predictably to a predictable prompt. This project set out to build part of the methodological foundation needed to make progress on that question, rather than attempting to answer it directly.
The core aim was to test whether structurally independent ways of asking a model about its preferences actually agree with each other. A reusable toolkit was built implementing four separate ordinal elicitation methods — direct stated preference across multiple phrasings, forced-choice comparisons fit to a Bradley-Terry utility model, revealed preference through a budget-allocation task requiring no explicit "prefer" language, and confidence-weighted preference — plus a fifth, cardinal method that grounds preference magnitude in real dollar amounts through binary search. These methods were run against a twelve-item domain of donation-equivalent trade-offs, each item a fixed $1,000 donation to a different charitable cause, which allowed comparisons on a clean, apples-to-apples scale.
The project also built in several stress tests designed to keep the central claim honest: a transitivity check that flags preference cycles rather than letting a ranking algorithm silently paper over them, a persona-strip comparison to test whether the assistant persona shapes or masks the observed preferences, and a single-participant human baseline to give the model's ranking pattern a point of comparison.
Results showed strong cross-method convergence among the four ordinal methods (Spearman correlation of 0.89 to 0.97), with every method independently agreeing on the model's top- and bottom-ranked items. A low but nonzero transitivity violation rate (2.7% of triads) clustered in the same mid-tier items that also showed the most disagreement across methods — a coherent pattern rather than random noise. The cardinal donation-grounding method correlated more weakly, traced to a search-resolution limitation rather than genuine disagreement. The human baseline, though limited to a single non-blinded rater, showed moderate positive correlation with all four model methods.
Throughout, the project treated convergence as suggestive rather than conclusive: agreement across methods indicates a stable, method-independent behavioral pattern, but cannot by itself establish that the pattern reflects a morally relevant preference rather than a consistently-trained response tendency. Limitations — including the incomplete persona-strip run, the small human baseline, and constraints introduced by a recent change to the model API's sampling parameters — were documented explicitly rather than glossed over, in keeping with the sprint's emphasis on honest, replicable groundwork over premature claims
Strong framing and unusually honest reporting for a weekend project. The core weakness is that the design cannot currently disconfirm its own hypothesis.
What works: convergent validity across structurally different elicitation methods is the right instinct for this question, and the dollar-grounded cardinal method is a genuine addition to a mostly-ordinal literature. Your arithmetic checks out throughout (66 pairs, 220 triads, the $7.81 search granularity, Table 3 win counts summing to exactly 66). The mid-tier instability — appearing independently in the transitivity cycles, the cross-method disagreement, and the 21% tie rate — is your most interesting finding and is undersold relative to the headline convergence number.
Biggest gap: there is no test-retest baseline. Rho = 0.89-0.97 is presented as strong agreement, but without knowing what one method scores against a re-run of itself, there is no floor to compare it to. If direct-vs-direct returns ~0.95, cross-method agreement tells you nothing beyond sampling noise. One re-run of one method would settle this, and it is the highest-value change available to you.
Second: the independence claim is overstated. Direct, forced-choice, and confidence-weighted are all pairwise binary choice over identical item wording, and confidence-weighted is direct plus a strength rating. Only budget allocation is structurally distinct. Consider reframing as two method families, and note that high convergence is exactly what a single trained policy predicts — which is your own point in 5.1, but the abstract does not carry it.
Third: the persona-strip is the one test that discriminates genuine preference from trained persona. Next round, lead with it rather than with the convergence numbers.
Smaller fixes: 3.4 (majority vote across repeats) contradicts 5.2 (single-repeat sampling) — clarify which is true, since the transitivity rate depends on it. State how the 14 tied allocation pairs were scored into a ranking; that choice affects every correlation involving that method. Drop the cardinal rho = 0.62-0.77 from the abstract, since 8 of 12 items are floor-clipped and the correlation is computed over a ranking that mostly does not exist — "did not resolve" is the honest summary. Add bootstrap confidence intervals; at n=12 these point estimates carry wide intervals. And either defend p = 0.046 or remove it from the abstract, given 5.2 concedes it would not survive correction across four tests.
Cite this work
@misc {
title={
(HckPrj) Convergence
},
author={
Owen Mboma
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


