Three Ways to Ask a Model What It Is Doing, and How Little They Agree

- the postmortem section with then a full enumeration to find the actual null was a great section

- n=20 matched triplets provide enough statistical power for aggregate mean bounds but individual pairwise intervals still wide enough to cross the fair-agreement threshold so this can be expanded to larger n

- cassifies correlations between feature activations and conversational arms. Causal interventions (e.g., clamping identified SAE features or applying steering vectors during the neutral probe) are needed to establish links between internal states and downstream responses.

- Replicate the three-way instrument across open-weight models with public suites (e.g., Llama) to determine if near-zero agreement is a universal property

You built three genuinely different routes to the same target and measured how much they agree, using a statistic that accounts for chance agreement. You then reported the result as null, even though an earlier version of the analysis had suggested otherwise. The transparency made the paper much easier to trust.

Three places I would push further:

Run one arm again with a fresh seed and report how well each method agrees with itself. The title emphasizes the low agreement across methods, but it is hard to interpret that without knowing how much agreement is realistically achievable within a method. Since the reads are already on disk, this seems like a fairly accessible check.

Give the behavior arm the same sensitivity analysis you already did for the internal arm. You have already done the harder version of this analysis, and adding it here would help distinguish disagreement between instruments from ordinary measurement noise. I would also add the tail probabilities from the results table directly to the accuracy figure and mark which bars do not exceed chance. Right now, I think the figure makes the result look somewhat stronger than the statistical evidence supports.

Include the actual probe sentence and at least one survey item, along with the decoding settings. The paper is partly about how the route used to ask a question changes the answer, so it was surprising not to see any of the actual stimuli. Even half a page in the appendix would be enough.

For a follow-up, I would be interested in seeing the same three instruments applied to a setting where the prompt itself does not reveal the target state. You already point toward this limitation in the paper. Solving it would move the work from testing whether a manipulation was successfully reflected in the measurements toward a stronger test of whether these instruments recover something genuinely latent.