Keep Apart Research Going: Donate Today
Mar 10, 2025
A Noise Audit of LLM Reasoning in Legal Decisions
Mindy Ng, May Reese, Markela Zeneli
Summary
AI models are increasingly applied in judgement tasks, but we have little understanding of how their reasoning compares to human decision-making. Human decision-making suffers from bias and noise, which causes significant harm in sensitive contexts, such as legal judgment. In this study, we evaluate LLMs on a legal decision prediction task to compare with the historical, human-decided outcome and investigate the level of noise in repeated LLM judgments. We find that two LLM models achieve close to chance level accuracy and display low to no variance in repeated decisions.
Cite this work:
@misc {
title={
A Noise Audit of LLM Reasoning in Legal Decisions
},
author={
Mindy Ng, May Reese, Markela Zeneli
},
date={
3/10/25
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
