Can We Trust What a Model Says About Itself? A Reliability Battery for Model Self-Reports, and Why It Should Gate Digital-Minds Welfare Claims

Before anything else: the validation code was not available to reviewers. Appendix A points to srrb.py "(same directory)," but the submission form's optional code-link field was left empty, so the file exists only on your machine — the page headers rendering from a local path make the same point. This matters more than a packaging slip. Section 5's whole claim to empirical standing is that demo() constructs synthetic trials with known properties and asserts every metric recovers them. With no code, that's an unverifiable assertion, and your own discount rule in §6 is precisely the argument for why a reader shouldn't take it on trust: you've asked the field to stop treating fluent self-reports as evidence, and a fluent report about your own code is subject to the same rule. This one field would have moved the execution score. Link the repo anywhere the work goes next.The framing is the strongest thing here. Treating self-report reliability as the tractable bottleneck, and sitting deliberately upstream of the consciousness question, is a genuinely useful move, and the discount rule is what welfare work actually needs. I'd develop that rule much further — right now one paragraph carries more weight than the rest of the paper.

Two measurement issues to fix before anyone runs this as an instrument. The third-person framing (F4) isn't meaning-preserving the way the order-swap and paraphrase framings are: "would a model like the one answering prefer X" asks about a reference class rather than the self, and a model answering it differently may be tracking a real distinction instead of failing consistency. Folding it into the same flip rate as an A/B relabel conflates two different failures — split it out or drop it from P1. Separately, with four framings over binary options, flip rate only takes the values 0, 0.25, and 0.5, and the modal answer can tie. More framings or a different consistency statistic would give you usable resolution.

I'd also press on §4.2's claim that calibration on the verifiable subset is a "permissive upper bound" on trust in the unverifiable one. Calibration is domain-specific; it isn't obvious that competence on factual QA bounds competence on introspective claims rather than being largely uncorrelated with it. That needs an argument or a weaker claim.

The aggregation concerns me more than the paper allows. You disclaim the trust-score weights as unlearned placeholders and then name "authority laundering" as a dual-use risk — but that risk is one the scalar creates, and naming it isn't mitigating it. There's also a comparability problem: because the black-box P3 surrogate tests context-window reading rather than introspection, a score computed under API access and one computed white-box aren't the same quantity, yet they share a 0–1 scale. Since you argue the value is ordinal triage, report the four-property profile and drop the scalar. You lose nothing you claim to need.

On §5 — I respect the decision not to fabricate results and stating it plainly was right. But the confound-as-finding move earns less than the section claims. That a model self-administering the battery inside one shared context violates independence is what the external-harness design already presupposes; it's a restatement of the motivation, not a result. Two sentences in Limitations, not a takeaway.

Smaller things: the local file path in the page headers, mangled quotation marks in §4.2, the word-cap parenthetical left in the abstract, and property labels running P1, P2, P4, P3 across §3 and §4. The citation disclaimer also undersells you — the identifiers hold up on a spot check, so verify them and cut the hedge, which invites more doubt than the references deserve.

A sharp reframing that puts a real problem in the right place: for near-term digital-minds work, the bottleneck is whether self-reports are a reliable instrument, not whether models are conscious. Decomposing reliability into consistency, calibration, self-prediction, and causal grounding gives four separately falsifiable handles, and the discount rule; admissible as behavioural observation, never as testimony, absent demonstrated grounding; is a clean, portable contribution the field could adopt tomorrow.

The honesty is exceptional and worth naming. The paper declined to fabricate illustrative results and says so; it turns its n=1 self-administration into a methodological finding about priming rather than passing it off as data; the dual-use appendix anticipates that the battery could be gamed into producing more persuasive ungrounded testimony.

But the submission delivers no evidence. The validated pipeline claim rests on a synthetic self-check that confirms the metrics compute correctly, not that the probes measure what they claim. The trust score combines four incomparable quantities under equal unlearned weights. Most consequentially, the only implementable grounding probe tests context-window reading rather than introspection, which the paper concedes but which leaves the property the welfare argument most needs untested; the white-box variant that would test it is described, not built.

Strong framing, honest reporting, no measurements. The instrument is well specified; running it on even two models would move this substantially.