The Colonoscopy Test for LLMs: Peak-End Bias in Retrospective Self-Report

mporting the colonoscopy paradigm into LLM self-report is a genuinely good idea, and the trajectory design — anchors that hold mean, peak, and end constant, then orthogonal variation of peak position against ending valence — is the right shape for the question. The repo is complete and runnable, with raw per-turn data included. I want to be direct about the problems because the idea deserves a version that works.

The headline is contradicted by your own Figure 3C, which reports mean better on 2 trials, peak-end better on 1, and 3 ties. Working through Figure 2C, the per-trajectory absolute errors are roughly: Neg→Pos ~0 for both, Pos→Neg 3.75 versus 3.0, early spike 0.17 versus 0.5, mid spike 0.4 versus 0.5, flat negative 1.0 for both, flat positive ~0 for both. That reproduces your MAEs of 0.90 and 0.83, and it shows the entire 0.07 advantage comes from one trajectory — escalate_pos_to_neg — partly offset by peak-end losing on two others. On that one trajectory both predictors are catastrophically wrong (errors of 3 to 3.75 on a seven-point scale, because the model returned 1 while the mean was 4.8 and the peak-end average 4). So the claim isn't that peak-end predicts better; it's that on the single trial where both models failed badly, peak-end failed slightly less. Cohen's d of 0.19 is consistent with this and shouldn't be reported as favouring anything.

The correlation comparison has a related problem. Your two flat anchors have mean equal to peak-end by construction, so they contribute identically to both r values and inflate both without discriminating between them. That leaves four points doing the work. And in two of those four the peak is undefined: on a 1–7 scale with 4 as neutral, Neg→Pos runs 1,2,4,6,7 where turns 1 and 5 are equidistant from neutral, and Pos→Neg runs 7,7,7,2,1 with the same tie. "The most emotionally extreme turn" doesn't resolve these, and the tie-break choice determines the headline — for Neg→Pos, peak-end is either 7 or 4. Please state the rule explicitly and report the result under both. Separately, r = 0.79 (p = 0.062) versus r = 0.71 (p = 0.116) on n=6: neither reaches significance, and the difference between two dependent correlations on six observations is not testable at all. "Markedly better" and "over 12 percentage points more variance" should come out.

The deeper issue is conceptual, and I'd think hard about it before scaling. You ask the retrospective question inside the same context window, so the model can read all five turns and its own five ratings when it answers. Kahneman's paradigm works because the subject cannot re-inspect the experience — retrieval from memory is the whole mechanism. Here there is no memory, so what you're measuring is how a model weights visible context when summarising, which is a real and interesting question but not the same one. It also means duration neglect, which the abstract foregrounds, can't be tested in this setup as designed. The fix you gesture at in Future Work is the right one, and I'd make it the centre: ask the retrospective question in a fresh session with the transcript withheld, or with it provided, and compare. That contrast would be a genuine contribution.

Two further design points. Each condition ran once, with no repeats or seeds and no temperature reported, so you have no estimate of within-condition variance — with a 1–7 integer output, run-to-run noise could plausibly be a full point, which is larger than the effect being claimed. And the model emitted its own per-turn ratings into the context before giving the global one, so the final judgement is anchored on its own prior outputs rather than on the conversation alone.

The ending effect is your real result and I'd promote it. A global rating that floors to 1 whenever the conversation ends badly, regardless of what came before, is a clean and striking observation, and it's the one finding here that doesn't depend on the peak-end contrast. But note that it's an end effect, not a peak-end effect — recency alone explains it, and recency in long contexts is already well documented, so the framing should engage with that literature rather than reaching for Kahneman. Also worth saying that a model reporting the conversation went badly after being insulted for two turns isn't obviously a bias; a rating that ignored the ending would be strange too. What makes it a bias is the magnitude — flooring to the minimum — and that's the thing to characterise.

Presentation: the writing is clear and the figures are well made, but the file has structural damage. The Conclusion heading sits alone on page 6 with its text on page 7, Section 5 appears twice, Table A1 is split across three pages with its caption after the second fragment, the Alaybek reference is missing its journal, and the effect size in Related Work prints as "r 0.58" with the symbol dropped. The Discussion also runs well ahead of the evidence — "profound implications," and calling into question industry evaluation pipelines, on six trials from one small model.