Skip to content
Apart Research
Apart Research
Contact us
Sprint projectAug 17, 2026Taipei / Vancouver

Preference or Position? Auditing Pairwise Choice Readouts in Gemma-4-31B-it

Bo-Shen Chen, Yi-Wen Chu, Barry Yu, Andy Yu · Team Sacabam Bubble

Submitted to Digital Minds Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: Preference or Position? Auditing Pairwise Choice Readouts in Gemma-4-31B-it

Open in a new tab (opens in new tab)
Presentation

Presentation: Preference or Position? Auditing Pairwise Choice Readouts in Gemma-4-31B-it

Open in a new tab (opens in new tab)
Code (opens in new tab)
Share

LLM preference measurements are meaningful only if they are invariant to choices that should not matter. We measure preference by presenting a model with two outcomes and recording which one it picks, and we test two systems this way. In one open-weight model family and under a specific forced-choice prompt family, reported choices near the decision boundary were not invariant to presentation order. If similar effects occur elsewhere, they could affect pairwise preference benchmarks, model-as-judge evaluations, reward modeling, and attempts to infer model values.

Reviews

Judging this Sprint?

Review this project

Your public critique appears on this page without your name. Your private critique is not published; only the Apart team reads it. If you agree below, we share your review with grantmaking.ai (opens in new tab) and the Transformative AI Fund so strong projects can be funded.

Not shown on this page.

Shown on this page, without your name.

Only the Apart team reads this, and funders if you agree below.

Share my name publicly on grantmaking.ai *
Share my private critique with funders *

How much would this matter for the field if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.

  1. * One of the best executed papers I have read recently and the most immediately useful.

    Strengths:

    The confident-flip discovery: It provides a new way to see what standard aggregation does wrong - Near a tie, the model doesn't waver; it's over 99% sure the second-listed option is the answer, whichever option that happens to be.

    Areas to improve:

    * Every result comes from a single model. There could be a broader scope. But to their credit, they make this very clear in the title of the paper itself.

    * Expand more on why the bias points the wrong way: "Prior work usually finds the models favors the first option" -- I wish the paper went more in-depth into position bias.

  2. Your discipline about validity is the best part of this work. You validated the Utility Engineering estimator against planted preferences before you trusted it, and you established a repeatability floor. You also ran layout-against-label controls that separate position from answer token. You then invalidated your own logit-lens "decision trajectory" story instead of selling it. That self-refutation, with your refusal to treat 44 answers as 44 independent samples, is the epistemics this area needs. Two things limit the reach. Your headline effect rests on one scenario family, and on 14 independent scenario pairs in effect. The striking 44 of 44 therefore gives much less evidence than the number suggests. You also never engage the existing literature on position bias in multiple-choice evaluation and judge evaluation. A reader therefore cannot tell what is new here. Position that framing explicitly, and run even two more model families, and this becomes a citable measurement-validity result.

    Read full reviewShow less

Cite this project

@misc{chen2026preference,
  title = {{Preference or Position? Auditing Pairwise Choice Readouts in Gemma-4-31B-it}},
  author = {Bo-Shen Chen and Yi-Wen Chu and Barry Yu and Andy Yu},
  year = {2026},
  month = aug,
  note = {Submitted to Digital Minds Research Sprint, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/preference-or-position-auditing-pairwise-choice-readouts-in-gemma431bit-lapr}},
  url = {https://apartresearch.com/sprints/projects/preference-or-position-auditing-pairwise-choice-readouts-in-gemma431bit-lapr}
}

More from Digital Minds Research Sprint

All 233 entriesBrowse all projects

Build something like this at the next Sprint

AI Collusion Research Sprint · Oct 23 - 25, 2026