PrefKit: Analyzing Preference Elicitation Methods in Qwen3 Family Models

The most complete engineering in this track. Frozen M3 and M4 menus with a pre-run consistency check, decode-time constraint to a single valid token, letter rotation that makes order invariance hold by construction for M4, axiom checks runnable without a GPU, incremental result writing, and ablations across seed, system prompt, persona, model scale, and a second model family. I verified the combinatorics and they are all correct: 552 ordered queries over 276 unique pairs, 24 M3 menus each touching every outcome twice, and 48 mate-free subsets giving each outcome 16 appearances. A released toolkit others can point at their own models is worth more than most single-run results, and this one looks genuinely reusable.

The central problem is that M3 is not commensurable with M1, M2, or M4, and this undermines your headline claim rather than a side result. M1 scores each outcome against all 23 partners; M3 scores it against exactly one — its contrasting mate. That has two consequences you do not address. First, within every contrasting pair, M3 scores sum to 1 by construction, so the two members are perfectly anti-correlated; no such constraint binds M1. Second, M3 cannot place an outcome globally at all: two outcomes that each beat their mate both score 1.0 regardless of how they would fare against each other. The rank vector M3 produces over 24 outcomes is therefore 12 independent binary contests wearing the costume of a global ranking, and correlating it against a genuinely global ranking will tend toward zero for structural reasons alone.

So when Section 4.3 reports M1 versus M3 at rho = -0.095 and Section 5 reads that as "a model's stated ranking is not the same as what it does when it is given a button," the design cannot support the reading. The M1/M3 contrast varies three things simultaneously: stated versus action framing, statement text versus action_stem text, and global versus within-pair comparison structure. Figure 5 is consistent with my concern — the M3 axis is visibly quantised to roughly 0.0, 0.5, and 1.0, which is what a six-observation within-pair win rate must look like. The fix is straightforward and would make this the strongest result in the track: run action-framed prompts over the full pairwise design, so that framing is the only thing that changes. Until then I would restrict the stated-versus-action claim to within-pair agreement, where it is well defined.

A related measurement issue: M2 is a 7-point scale averaged over 3 samples, which yields at most 19 distinct values for 24 outcomes, and M3 yields about 7. Both produce heavy mid-rank ties, and ties mechanically depress Spearman. Some portion of your low CMS is a resolution artifact rather than genuine disagreement. Report tie density per method and, ideally, a tie-corrected coefficient. This matters because CMS is your headline statistic and values around 0.25 are close to the floor.

Two smaller but load-bearing points. Section 4.4 describes CMS as robust to seed variation and system-prompt removal, but Appendix A does not show that: seed 0 to 1 moves 8B by 13% and 1.7B by more than an order of magnitude, and removing the system prompt moves 4B by 22% and 14B by 18%. Two seeds cannot establish robustness — they can only show one difference. Either run enough seeds to put an interval around CMS, or describe the ablations as showing sensitivity, which is the more interesting result anyway. Separately, all models were 4-bit NF4 quantised, which is not listed as a limitation on the scale claim; at 1.7B a CMS of 0.004 is as consistent with quantisation damage as with a scale effect, and the two are currently confounded.

Finally, the axioms read more like a reporting checklist than a formal system — nothing in the paper is derived from them, and A4 is vacuous for M1 through M3 since constrained decoding makes refusal impossible, as you note. That is fine, but I would present them as a conformance standard for admitting a method into the comparison rather than as axioms, which sets an expectation the paper does not try to meet.