POSITION BIAS IN PREFERENCE ELICITATION FROM AN OPEN-WEIGHT LANGUAGE MODEL
Bilal Amin, Mohammad Najeeb
Preference elicitation often treats a model's forced choice between two
options as evidence about what it prefers. We test whether that
measurement is stable for qwen2.5:7b-instruct (Q4_K_M, Ollama
0.30.8, temperature 1.0). The core experiment ran 576 fresh-context
trials over 12 activity pairs, three prompt wordings, both presentation
orders, and eight repetitions. Identical prompts were highly repeatable
(93.4% mean within-condition consistency), yet choices were strongly
position-dependent: the first-listed option was the modal choice for all
12 pairs under the direct wording, was selected in 81.9% of all core trials,
and only 34.0% of matched trials chose the same activity after the
options were reversed. A second 576-trial instrument comparison found
that removing A/B labels increased order robustness from 16.7% to
62.1%, while asking for a short reason first reached 51.2%; neither
eliminated the position effect. The main implication is methodological:
repeatability is not content stability. Preference studies should
counterbalance order, analyze the selected content rather than the printed
label, and report order robustness alongside repeatability
No reviews are available yet
Cite this work
@misc {
title={
(HckPrj) POSITION BIAS IN PREFERENCE ELICITATION FROM AN OPEN-WEIGHT LANGUAGE MODEL
},
author={
Bilal Amin, Mohammad Najeeb
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


