Nov 23, 2025
MedOmitDetect: Ensuring Safety in Patient-Facing Medical LLM
Sonal Joshi
Large language models (LLMs) show promise for patient-facing medical chatbots, but their deployment raises critical safety concerns. While existing evaluation frameworks focus on detecting factual inaccuracies (hallucinations), they often overlook a more insidious risk: omissions—missing critical information that patients cannot independently identify. We introduce MedOmitDetect, an automated evaluation system that assesses the completeness of medical AI responses by detecting clinically significant omissions and classifying them by potential patient harm severity (Mild, Moderate, Severe, Life-threatening). Built on the clinician-annotated MedExpert dataset & benchmark experiments, our system provides a two-stage pipeline for generating and evaluating medical LLM responses across 8 models, enabling systematic safety benchmarking before clinical deployment.
No reviews are available yet
Cite this work
@misc {
title={
(HckPrj) MedOmitDetect: Ensuring Safety in Patient-Facing Medical LLM
},
author={
Sonal Joshi
},
date={
11/23/25
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


