Where Did That Confidence Come From?
Rohit Kale , Pranjal Panghal
One way to make claims about model metacognition more testable is to give a model a task whose answer requires causally controlled information from its own hidden computation and cannot be recovered from the visible transcript. We do this by perturbing a hidden confidence state in Gemma 3 while keeping the visible question and answer unchanged, then asking the model to report the hidden intervention through a counterbalanced code. The confidence direction has far greater causal efficacy at the answer-adjacent token than one token later and produces the strongest delayed readout among 40 matched directions. Yet an intervention isolated to Answer A’s cache row also influences a later confidence report about Answer B. The project therefore demonstrates functional access to hidden confidence information while showing that detection alone does not establish where that information originated.
This is an interesting study with a convincing methodology. On reading the presentation, this reader got lost in the details and could at many places not really follow. One thing that did not become fully clear is how your study differs from [7]:
“Kumaran et al. [7] provide a task-native state for this test. They showed that Gemma 3 27B forms a representation of answer quality immediately after answering, stores it in the key/value cache, and later retrieves it when reporting verbal confidence. We build on that causally validated state rather than injecting an unrelated concept.”
To highlight the novelty of your study, you write:
“Our study differs in two ways. The target is a naturally formed, independently characterized confidence state rather than an arbitrary semantic concept, and the primary specificity comparison uses 39 directions built by relabeling the same source activations. The row-isolation audit then tests an assumption that intervention-report paradigms usually leave implicit: whether the causal trace remains attributable to the state where it was introduced.”
If I understand correctly, [7] show *that* interfering with the representation of answer quality affects verbal confidence reports, whereas you show that confidence reports are not just affected by this, but are directly based on the representation of answer quality? If not, what exactly did you show? Either way, I recommend clarifying this part of the presentation.
The methods and results sections use many abbreviations and technical details that I cannot follow. If everyone working on this kind of research understands this, that’s fine, but readers who are not directly involved in this kind of research (like myself) are unlikely to understand this part.
The discussion and remarks about limitations seem reasonable and plausible.
- Authors do KV editing and show that: edit confidence trace "attached to A → later confidence query about A changes" but also that "edit confidence trace attached to A → later confidence query about B changes". This is interesting!
- The empirical tests are well done and have necessary controls (blocking attention as mentioned would be nice!).
- Understanding how confidence is represented, used, and bound to the particular content it is confidence about is interesting and important.
Cite this work
@misc {
title={
(HckPrj) Where Did That Confidence Come From?
},
author={
Rohit Kale , Pranjal Panghal
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
