Does the Elicitation Method Change the Preference We Measure?

The most statistically disciplined submission I reviewed in this track. You pre-specified the comparison family, applied Holm correction across all nine, reported a null, and declined to claim three raw p-values below 0.05 as independent evidence when they trace to one underlying pattern. Section 4.2 goes further and identifies the confound that partly explains your own most favorable result — that an INDIFFERENT answer is trivially positionally consistent, so part of the consistency gain is opting out rather than choosing more stably. Volunteering that is the mark of someone doing this properly. I also verified your arithmetic end to end: all nine mean differences reconcile with the bar values, 21 of 90 is 23.3%, both strength distributions sum to 90, and the implied means of 4.233 and 4.333 match your reported 0.10 difference.

You are underselling your actual finding. GPT-OSS-120B's forced-choice positional consistency is 53.3%, which is 24 of 45 — statistically indistinguishable from a coin flip against a 50% chance floor. That is not "lower reliability than other methods." It means that for this model, on this item set, forced-choice order-reversal carries essentially no information. Frame it against the chance baseline with a binomial test rather than only against the other two methods, and lead with it. It is a stronger and more useful claim than the method-spread framing, and it does not depend on any comparison surviving correction.

The main threat to the headline model-dependent claim is a configuration confound. GPT-OSS-120B ran at reasoning effort "low" and Qwen3.6-27B at "none". The between-model difference is therefore confounded with reasoning configuration, and a reader cannot tell whether they are seeing a model effect or an effort effect. Section 6 does not list this. Either match the settings and re-run, or state plainly that the cross-model comparison is confounded — the within-model method comparison for GPT-OSS-120B stands regardless and is the more defensible half of the paper.

Two reanalyses you can run for free on data you already have. First, recompute positional consistency for the explicit-indifference method after excluding responses answered INDIFFERENT. You correctly flag that this artifact inflates the number; excluding it converts the flag into an answer, and would tell you whether the method genuinely stabilises choices or merely offers an exit. Second, stratify by pair. You note in 4.4 that some pairs have a clear intended answer, which means positional consistency is partly a measure of pair difficulty rather than method quality. Splitting easy from ambiguous pairs would show whether the method effect concentrates where it should.

Smaller points. The strength scale is uninformative and that is itself a reportable result: neither model ever used 1 or 2, and GPT-OSS-120B used only 4 and 5. A self-reported strength scale that compresses into its top two points cannot function as a graded signal, which matters for anyone planning to use strength ratings downstream. Section 3.3 does not say how you paired the three original repetitions with the three flipped ones when computing consistency; with stochastic sampling the pairing rule affects the statistic, so state it. Figure 1 needs confidence intervals — at 45 comparisons per bar, 53.3% carries roughly a 15-point interval, which changes how the 33-point spread reads. Finally, Holm across all nine jointly is conservative given that the method comparisons and the model comparisons answer different questions; reporting both the joint and the split-family corrections would be more informative than choosing one.