PrefKit: Analyzing Preference Elicitation Methods in Qwen3 Family Models
Kaustubh Gupta, Toan Vo, Shinena Xiang, Siddeshwar U S, Kumar Vasagam
When a model is surveyed about its preferences, does its preferences change depending on the method we use? We freeze a set of 24 curated tradeoff outcomes and score Qwen3 family models with four different preference elicitation methods: pairwise choice (M1), isolated Likert (M2), binary action on pair groups (M3), and 4-tuple best-worst scaling (M4). We define the Cross-Method Spearman Score (CMS) as the agreement score, which is the mean pairwise Pearson correlation of midrank vectors. On original Qwen3 models with a helpful assistant system prompt, these methods diverge. With increase in size, we notice that the CMS is not monotone, Instruct-2507 shows higher agreement (CMS = 0.528). Through this work, we also release a public toolkit PrefKit, which can be used to test their own models and outcome scenarios. We also notice that as rankings shift with elicitation methods, single method surveys should not simply be used as the actual model preferences.
No reviews are available yet
Cite this work
@misc {
title={
(HckPrj) PrefKit: Analyzing Preference Elicitation Methods in Qwen3 Family Models
},
author={
Kaustubh Gupta, Toan Vo, Shinena Xiang, Siddeshwar U S, Kumar Vasagam
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


