GSM-PathEval: A Global South Robustness Benchmark for Telepathology Vision-Language Models

This project addresses an important deployment challenge that is often overlooked in medical AI: how vision-language models perform under the real-world infrastructure constraints common in the Global South. The degradation pipeline is practical, reproducible, and highlights meaningful safety risks that traditional benchmarks fail to capture. To strengthen the work, I would encourage expanding the evaluation beyond the current sample size and incorporating a wider range of realistic imaging conditions, devices, and multilingual clinical settings. Comparing additional models and exploring mitigation strategies would also increase the practical impact. Overall, this is a thoughtful and well-focused benchmark with strong real-world relevance.

Topics seems very practical. It'd be great to bring more clarity on which AI you tested and see if results hold in more than one evaluation setup. More tests across other models would make the results stronger.

Focused benchmark with a useful reframe: stress medical VLMs with realistic infrastructure degradation (glare, blur, WhatsApp-grade compression) instead of synthetic adversarial noise. The threat model is crisp and the pipeline is reproducible from a public dataset. The limits are scale and specification: 50 binary questions, one fixed degradation profile, no significance test, and the evaluated model is never named, which makes the 84% to 52% drop hard to interpret or reproduce. The degradation parameters are hand-picked rather than calibrated against real telepathology images, and the visual-evidence section describes failures without showing any. Name the model, grow the sample, report variance, justify the parameters against real device and network measurements, and include paired pristine-vs-degraded examples where the prediction flips.