Having a State Is Not Knowing It

The five-gate hierarchy is highly useful, and other researchers should adopt it. The authors built trust by avoiding easy-to-cheat testing methods. However, the core finding has a small margin of error and flips with new data mixtures. To strengthen the paper, the authors should add repeated tests, confidence intervals, and open-source data. Finally, a clear flowchart and simpler formatting would make it easier to read

This work asks a very interesting question: whether language models can predict how hypothetical interventions to their own internal states would affect their behavior. The experimental design is thoughtful, though a schematic figure of the intervention, prediction, and scoring setup would make it much easier to follow.

One concern is that poor introspective performance after an intervention may not necessarily indicate limited introspective ability, since the steering itself could disrupt the computation needed for introspection or accurate reporting. A related concern also appears in the cited work. It would be interesting to test whether allowing the model additional computation or reasoning before reporting mitigates this effect.

The limited OOD generalization also leaves open whether the observed failures reflect a fundamental limitation or insufficient training, data, or model capacity; scaling these factors would help distinguish the two. Finally, the weak prediction of quantitative effect size may partly reflect the difficulty of mapping internal changes onto an abstract continuous magnitude. A simpler ordinal scale (e.g., no effect / small / medium / large effect) might provide a more natural test of whether the model can estimate intervention strength.