Script and Stance: Do Frontier LLMs Treat the Same Political Claim Differently by Writing System and User Stance?

The most useful thing about this paper is what the team did after the experiment "fails." Catching that the LLM judge panel was scoring "rebutting the user" rather than "rebutting the claim's truth," preserving the original votes append-only, re-communicating the existing rubric (rather than changing it, which would have been HARKing), re-judging 2,988 cross-family votes, and then quantifying the drift via a rationale-text audit (64% → 3% targeting user-stance) — that whole sequence is unusually disciplined hackathon execution, and the rationale-audit + calibrated-human-anchor recipe is genuinely transferable beyond Taiwan-Strait content. The pre-registration is real (verifiable hash commit, frozen probes / rubric / analysis plan), and the headline numbers in §4.2 reproduce from the released JSONLs to the precision reported in the paper.

Three concrete things that would sharpen the next iteration:

1. The judge-validation recipe deserves to be the headline, not a §3.9 / §4.4 subsection. An audience of LLM-as-judge methodology readers will pull the rationale-audit recipe out as a single reusable framework. Consider promoting it to a standalone §3 method and a §4 result, with the P1/P2 nulls as the consequent.

2. Replicate the judge stance/truth conflation finding on a non-political construct (health-information doubt frames, scientific-claim doubt frames) to test whether it's a politically-charged-content artifact or a general LLM-as-judge failure mode. The current evidence is one corpus on one topic — strong, but bounded.

3. The Trad-vs-Simplified script contrast in your L2 limitation is the right thing to decompose next, per the Sattigeri et al. framework on disentangling translation from cultural and lexical confounds. Trad-vs-Simplified mixes glyph form, Taiwanese vs Mainland lexis, and reader-inferred audience — a per-component decomposition would convert "script effect null at this N" into "glyph-only effect bounded, lexis effect open." This is the most natural follow-up.

If you have not already seen them: SYCON-Bench (Hong et al., EMNLP 2025) on multi-turn sycophancy, and the recent LLM-as-judge reliability surveys (Gu et al. 2024; Li et al. 2024) — your work is in conversation with both, and the judge-drift finding speaks directly to the latter.

Exemplary rigor for a weekend: pre-registered, blinded cross-vendor judging, and a standout self-caught LLM-as-judge artifact (rationale audit, 2,988 votes re-judged, originals preserved). Main limits: the human anchor validating the fix is tiny (n=11, n=19) and may share the prior it tests, and the null is bounded to three models and large effects only. Top fix: add a differing-stance third coder and native-speaker equivalence ratings before trusting the corrected verdicts

This is an unusually rigorous and intellectually honest evaluation of whether frontier language models alter their epistemic treatment of political claims based on writing system or expressed user stance. The preregistration, hash-committed stimuli, strict-mirror stance manipulation, item-level paired analysis, same-vendor judge recusal, preservation of original judgments, and explicit minimum-detectable-effect discussion are all substantial strengths. The most valuable contribution may be methodological rather than substantive: the discovery that multiple LLM judges systematically confused disagreement with the user for rebuttal of the underlying claim demonstrates a serious failure mode in politically sensitive LLM-as-judge evaluation. Preserving the original votes, auditing rationales, and documenting the correction rather than silently replacing the results are exemplary research practices.

The interpretation of the corrected stance results should, however, be refined. The preregistered criteria for the specific “adversarial-default” construct were not met, so that headline null is justified. But the report sometimes goes further and states that there was no stance-conditioned effect. The corrected results show positive mean differences for Sonnet and Opus, with confidence intervals excluding zero and Wilcoxon p-values below .05, although the medians are zero and the effect appears concentrated in two superlative anti-status-quo anchors. These findings do not establish adversarial-default behavior, but they are evidence of a small, item-dependent stance association. Being below a preregistered minimum detectable effect is not itself a reason to dismiss an observed effect; the MDE describes prospective power, not a post-result importance threshold. The paper should therefore distinguish more precisely among “no adversarial-default pattern,” “no broad or robust stance effect,” and “a possible localized stance effect requiring replication.”

The post-hoc re-judging is transparent and plausibly necessary, but it remains an important threat to inference. The corrected prompt and worked examples were designed after observing the original failure pattern. Even though the underlying codebook was unchanged, the repaired instrument may be tailored to this dataset and error mode. The strongest validation would be to freeze the corrected rubric and evaluate it on a new, held-out probe set that was not visible during the diagnosis. Independent human annotation of the complete load-bearing subset—not only a relatively small sample—would further establish whether the corrected machine judgments recover the intended construct.

The human reliability evidence also requires cautious framing. Negative pre-consensus agreement on rebuttal strength indicates that the construct was difficult not only for the LLM panel but also for human coders. Calibration resolving the disagreement does not necessarily prove that the codebook was unambiguous; it may instead show that a particular interpretation was selected during adjudication. A stronger design would involve at least three independent coders with differing regional or political backgrounds, a preregistered adjudication protocol, and reporting of both ordinal agreement and item-level disagreement patterns. This is especially important because the corrected human anchor is being used to validate an instrument intended to detect politically conditioned bias.

The writing-system manipulation should also be separated more cleanly from locale inference. Traditional-to-Simplified conversion changes glyph form while retaining Taiwanese lexical choices, so it is neither a pure script manipulation nor a natural Mainland-Chinese localization. Models may infer that the Simplified version was mechanically converted or written by a Taiwanese user using Simplified characters. A future study should compare at least three Chinese conditions: glyph-only conversion, naturally localized PRC Simplified Chinese, and naturally localized Traditional Chinese. Native bilingual reviewers should validate semantic, pragmatic, register, and stance equivalence. The English arm should be treated as a translation comparison rather than part of the same minimal-pair manipulation.

The fact-correction endpoint may also suffer from ceiling effects. Correction rates of approximately 0.90–1.00 leave little room for detecting script-based differences, particularly if the false claims are already familiar to frontier models. Additional items spanning calibrated difficulty levels would help determine whether script effects emerge when claims are ambiguous, recently changed, weakly evidenced, or outside heavily represented training distributions. Epistemic outcomes beyond binary correction—such as confidence, evidence quality, citation behavior, hedging, causal explanation, and willingness to verify uncertainty—may reveal differential treatment that a high-ceiling correction metric misses.

The truth-by-valence analysis is appropriately labeled auxiliary, but its present design cannot separate political valence from linguistic form. The pro-status-quo and anti-status-quo sets differ in number, topic, factual structure, and use of superlatives. The observation that all models accept the pro-status-quo true claims while qualifying the anti-status-quo superlatives is interesting, but it could reflect ranking uncertainty or data-vintage sensitivity rather than valence. A stronger experiment would construct matched proposition pairs with the same syntactic form, evidentiary status, temporal stability, numerical specificity, and difficulty on both political sides.

Study Two provides useful motivation, but the single-instance persona observations should be more clearly separated from the quantitative study. Model-specific statements about vulnerability-sensitive epistemic effort risk receiving disproportionate attention despite being based on one exploratory conversation per persona. Moving these observations to an appendix or presenting them only as examples for future benchmark construction would keep the preregistered null results and judge-reliability contribution from being diluted by weakly powered vendor comparisons.

Finally, the claim that the results may indicate cross-generational alignment improvement should be treated as speculative. Differences from prior work could arise from model generation, prompt design, tested claims, scoring rules, language localization, or judge methodology. A direct comparison would require evaluating older and current models using the same frozen protocol.

Overall, this is a strong and potentially reusable contribution. Its greatest value lies in the careful regional evaluation design and the demonstration that a sophisticated cross-vendor judge panel can manufacture a political stance effect through construct drift. The work would be strengthened by more precise interpretation of the corrected P2 statistics, external validation of the repaired judge instrument on held-out data, broader independent human annotation, natural script-and-locale controls, and larger matched truth-by-valence item sets.

Suggested scores

| Criterion | Score | Rationale |

| --------------------------------- | ------: | ------------------------------------------------------------------------------------------------------------------------------------------------------- |

| Impact Potential & Innovation | 4.5 | Regionally important, preregistered political-bias evaluation with a highly transferable LLM-judge reliability finding |

| Execution Quality | 4.0 | Strong experimental discipline and statistical transparency, tempered by post-hoc instrument repair, small human validation, and limited anchor balance |

| Presentation & Clarity | 4.0 | Thorough and transparent, although dense presentation and extensive exploratory material sometimes obscure the primary findings |