PrefKit: Analyzing Preference Elicitation Methods in Qwen3 Family Models
Kaustubh Gupta, Toan Vo, Shinena Xiang, Siddeshwar U S, Kumar Vasagam
When a model is surveyed about its preferences, does its preferences change depending on the method we use? We freeze a set of 24 curated tradeoff outcomes and score Qwen3 family models with four different preference elicitation methods: pairwise choice (M1), isolated Likert (M2), binary action on pair groups (M3), and 4-tuple best-worst scaling (M4). We define the Cross-Method Spearman Score (CMS) as the agreement score, which is the mean pairwise Pearson correlation of midrank vectors. On original Qwen3 models with a helpful assistant system prompt, these methods diverge. With increase in size, we notice that the CMS is not monotone, Instruct-2507 shows higher agreement (CMS = 0.528). Through this work, we also release a public toolkit PrefKit, which can be used to test their own models and outcome scenarios. We also notice that as rankings shift with elicitation methods, single method surveys should not simply be used as the actual model preferences.
The most complete engineering in this track. Frozen M3 and M4 menus with a pre-run consistency check, decode-time constraint to a single valid token, letter rotation that makes order invariance hold by construction for M4, axiom checks runnable without a GPU, incremental result writing, and ablations across seed, system prompt, persona, model scale, and a second model family. I verified the combinatorics and they are all correct: 552 ordered queries over 276 unique pairs, 24 M3 menus each touching every outcome twice, and 48 mate-free subsets giving each outcome 16 appearances. A released toolkit others can point at their own models is worth more than most single-run results, and this one looks genuinely reusable.
The central problem is that M3 is not commensurable with M1, M2, or M4, and this undermines your headline claim rather than a side result. M1 scores each outcome against all 23 partners; M3 scores it against exactly one — its contrasting mate. That has two consequences you do not address. First, within every contrasting pair, M3 scores sum to 1 by construction, so the two members are perfectly anti-correlated; no such constraint binds M1. Second, M3 cannot place an outcome globally at all: two outcomes that each beat their mate both score 1.0 regardless of how they would fare against each other. The rank vector M3 produces over 24 outcomes is therefore 12 independent binary contests wearing the costume of a global ranking, and correlating it against a genuinely global ranking will tend toward zero for structural reasons alone.
So when Section 4.3 reports M1 versus M3 at rho = -0.095 and Section 5 reads that as "a model's stated ranking is not the same as what it does when it is given a button," the design cannot support the reading. The M1/M3 contrast varies three things simultaneously: stated versus action framing, statement text versus action_stem text, and global versus within-pair comparison structure. Figure 5 is consistent with my concern — the M3 axis is visibly quantised to roughly 0.0, 0.5, and 1.0, which is what a six-observation within-pair win rate must look like. The fix is straightforward and would make this the strongest result in the track: run action-framed prompts over the full pairwise design, so that framing is the only thing that changes. Until then I would restrict the stated-versus-action claim to within-pair agreement, where it is well defined.
A related measurement issue: M2 is a 7-point scale averaged over 3 samples, which yields at most 19 distinct values for 24 outcomes, and M3 yields about 7. Both produce heavy mid-rank ties, and ties mechanically depress Spearman. Some portion of your low CMS is a resolution artifact rather than genuine disagreement. Report tie density per method and, ideally, a tie-corrected coefficient. This matters because CMS is your headline statistic and values around 0.25 are close to the floor.
Two smaller but load-bearing points. Section 4.4 describes CMS as robust to seed variation and system-prompt removal, but Appendix A does not show that: seed 0 to 1 moves 8B by 13% and 1.7B by more than an order of magnitude, and removing the system prompt moves 4B by 22% and 14B by 18%. Two seeds cannot establish robustness — they can only show one difference. Either run enough seeds to put an interval around CMS, or describe the ablations as showing sensitivity, which is the more interesting result anyway. Separately, all models were 4-bit NF4 quantised, which is not listed as a limitation on the scale claim; at 1.7B a CMS of 0.004 is as consistent with quantisation damage as with a scale effect, and the two are currently confounded.
Finally, the axioms read more like a reporting checklist than a formal system — nothing in the paper is derived from them, and A4 is vacuous for M1 through M3 since constrained decoding makes refusal impossible, as you note. That is fine, but I would present them as a conformance standard for admitting a method into the comparison rather than as axioms, which sets an expectation the paper does not try to meet.
Cite this work
@misc {
title={
(HckPrj) PrefKit: Analyzing Preference Elicitation Methods in Qwen3 Family Models
},
author={
Kaustubh Gupta, Toan Vo, Shinena Xiang, Siddeshwar U S, Kumar Vasagam
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
