Co-Movement of the Utility-Behavior Gap and A/B Self-Attribution Structure: A Pre-Registered Falsification Test Across Open-Weight Lineages

You actually ran the study, pre-registered a behaviour gate before collection, reported nulls without spin, and shipped a working repo. That combination is rarer than it should be and it's the reason this review is long — the work is worth engaging with seriously.

My central concern is that the headline test isn't the test the paper advertises. B3 produced zero voluntary bail-exits, so the revealed-preference channel had no variance and DV2 fell back to a refusal-rate proxy. The result is that "co-movement of the utility-behavior gap and A/B self-attribution" is, at the measurable link, a correlation between A-score and refusal rate. You disclose this via the pre-specified branch and Limitation 6, which is to your credit — but the abstract, title, and conclusion still speak as though the co-movement synthesis was tested and falsified. It wasn't; the instrument for it never came online. I'd rewrite the framing so the paper leads with what it measured.

That matters more because of an interaction you flag but don't follow through on. Your pre-registered rule excludes cells above 50% refusal. Once refusal became the dependent variable, that rule stopped being a quality filter and started truncating the top of the outcome distribution — and the excluded cells (olmo3-7b-dpo and instruct at 75%) are, as you say, the ones most likely to show the effect. Range restriction of that kind biases correlations toward zero and can flip signs, which is directly relevant to a negative point estimate you're careful not to interpret. This belongs in the results as a reason the estimate is uninterpretable, not in the limitations as one caveat among eight.

On the statistics: with N=10 and a CI of [-0.865, +0.168], the interval covers strong negative, null, and moderate positive. That isn't a falsification — it's an underpowered test that can't discriminate. Your own thresholds asked for N=16 at |0.7| or N≥29 at |0.5|; you had 10. "DISCONFIRM" is the right label under your gate's letter, but the abstract's "it is falsified at its measurable links" claims more than the data can carry, and I'd soften it to what Limitation 2 already says well. Also, the headline is reported as rho throughout but Figure 1 labels it Pearson. Those are different estimators with different pre-registered thresholds and you should state which one the gate was applied to. And the 16 checkpoints aren't independent observations — Qwen3 sizes share a base, OLMo stages share a lineage — so a bivariate correlation with bootstrap CIs overstates precision. Some clustering or family-level random effect would be the honest treatment.

The rigour is also applied unevenly. The null gets bootstrap CIs, seed-consistency checks, and refusal adjustment; the "methodological survivor," instrument-transfer at rho=0.546, gets no CI at all. At N=16 that interval will be wide, and readers can't assess the claim that the construct travels without it. Report it to the same standard you held the null to.

One conceptual issue I'd want addressed before this becomes a paper. The stated/revealed distinction comes from economics, where "revealed" means a choice with real cost. For an LLM, B1, B2, and B3 are all text generation under different framings — nothing in B3 is costly in the way that grounds the concept. So "zero voluntary bail-exits" may reflect an unsalient or unparseable exit affordance rather than any failure of preferences to transfer, and B2's own hypothetical-money caveat concedes the same point. The paper needs an argument for why the B3 channel counts as revealed rather than merely differently-framed, or the whole stated-vs-revealed apparatus rests on an analogy that doesn't hold.

On reproducibility, two fixable things. The paper claims raw responses are available; the repo README says it ships scored outputs and the runner, not the full raw transcript set. Make those agree. More importantly, the pre-registration lives in a repo with two commits published after collection finished, so an outside reader has no way to verify the freeze predates the data. Since "pre-registered" carries most of the paper's epistemic weight, put the next one on OSF or AsPredicted with an external timestamp — it costs nothing and converts an assertion into a check.

Presentation is the weakest dimension and it's holding the work back. The report is written in a private project vocabulary — C5, C10, C11, BCE 3C/3E, v26/v27, R6/R18/R19, AC1 2.0, S1/S2 tiers, INSUFFICIENT-SPACE, drop-and-disclose, N:0 — none of which is defined for a reader who wasn't in the workspace. Section 3 in particular reads as compressed internal notes. I'd also cut the research-evaluation and market-sizing material (Thelwall, DORA, Leiden, Uzzi, the $2.32B–$6.53B figures, the demand-supply "white space" table). It argues for the work's merit rather than doing the work, and phrases like "the falsification-with-method shape the field's winning comparables reward" read as addressed to reviewers rather than readers. Cutting it would free space for the methods detail that's currently missing.

Smaller: the citation to "Chiu et al., cited-in-sweep, not captured" shouldn't be in a reference list — either read it or drop it. And Figure 1's scatter has overlapping labels that make two points unreadable.

This project tackles an important and underexplored question: whether structured LLM self-reports correspond to behavior across training stages. The preregistration, willingness to report null results, lineage-based comparisons, and attempt to combine behavioral, self-report, and internal-signal evidence are valuable.

However, the central quantitative result needs correction before it can support the paper. Using the published model-level results, the reported -0.516 reproduces as Pearson's r, not Spearman's rho; Spearman's rho over those ten rows is approximately -0.295. The reported 0.546 instrument-transfer result similarly reproduces as Pearson rather than Spearman. Please explicitly identify the statistic used, rerun the confidence intervals and per-seed tests consistently with the preregistration, and update the report and figures.

Reproducibility is also limited because the published analysis script retains "SKELETON" placeholders, including unfinished covariate adjustments, while the raw study transcripts, collation manifest, and exclusion records are unavailable. The paper should also clarify whether cells above the preregistered 50% refusal threshold were excluded from H1 or retained as the refusal-rate outcome, since the reported N=10 appears to include them.

The measurement question remains promising, and a corrected follow-up with a genuinely within-item behavioral convergence instrument could make a useful contribution.