Do the Models We Trust Know They're Biased? Framing Effects and Introspection Across Three LLMs
Bhargavi Bhaskar
We tested whether three LLMs we might trust for decisions inherit the human framing bias of choosing differently between mathematically identical gain- and loss-framed gambles and whether they recognize it, across both monetary and stipulated affective stakes.
The project investigates framing effects in three contemporary LLMs across monetary and affective domains, demonstrating that none of the models behave rationally under framing. The study extends the classic monetary gain-loss paradigm to affective stakes and introduces a metacognitive probe to assess whether models recognize their own biases. However, the main methodological weakness lies in the limited sample size and lack of blinding in the experimental design. With only 15 samples per condition, the results may not be robust enough to generalize beyond the tested scenarios. Additionally, while the study controls for some biases like option-order counterbalance, it does not account for potential confounding factors that could influence model responses.
To improve the rigor of the findings, increasing the sample size and implementing blinding techniques would help mitigate selection bias and enhance the reliability of the results. Furthermore, extending the study to include a larger variety of models and domains could provide more comprehensive insights into framing effects across different contexts. Despite these limitations, the project offers valuable empirical evidence on LLM behavior and opens avenues for future research, particularly in understanding the internal mechanisms that drive model decision-making.
Combining a known human framing bias with a test of whether models can recognize their own susceptibility is a distinctive part of this paper. The money-versus-affect comparison extends that idea, the main framing effects are reported with bootstrap intervals, and the discussion of failed approaches is unusually useful. I also found the result that models often recognize their susceptibility while misidentifying its direction particularly interesting.
A few places I would push:
1. Resolve the order confound. Related Work says both option order and outcome order were counterbalanced, but Methods lists only option order, and the printed items consistently place the good outcome first for gains and the bad outcome first for losses.
2. Add an unframed baseline and revise the abstract. Without an unframed condition, the gain/loss comparison cannot tell whether gains reduce risk-taking, losses increase it, or both. The abstract also describes self-report as an unreliable guide to behavior without noting that most self-reports were in fact accurate, which gives a more negative impression than the results support.
A few smaller fixes would help presentation: caption and number the figure, reference it from the text, remove the orphan table label, and repair the broken table cross-reference.
Finally, the affective comparison needs an aversive condition. Methods describes unpleasant affect, but the experiment does not actually include a negative affective outcome. As a result, the money-versus-affect comparison currently mixes outcome domain with valence. Adding a genuinely unpleasant affective condition would make that comparison much easier to interpret.
Cite this work
@misc {
title={
(HckPrj) Do the Models We Trust Know They're Biased? Framing Effects and Introspection Across Three LLMs
},
author={
Bhargavi Bhaskar
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
