Doctorless

You're tackling a medical domain where accuracy matters for life-and-death decisions. Your benchmark for testing whether frontier LLMs can safely handle doctor-like tasks is the right problem. The fact that you're stress-testing on underserved use cases adds real value.

What hit me: you're building an evaluation platform, not just a dataset. That's harder and more useful. But I don't see inter-rater agreement reported on your medical expert annotations. High-stakes medical evals need Cohen's kappa or Fleiss' kappa to show annotator reliability. Did you measure model calibration on uncertainty? Doctors need to know when the model doesn't know.

Push next: get radiologists or clinicians to validate your rubrics independently before scaling. Medical benchmarks need domain expert sign-off.

Important problem and a sensible plan: 50 health cases sorted by how serious they are, to check if the AI sends people to real care. But the test wasn't really run — usage limits meant only a small pilot on one model, and the "findings" are mostly "it works." So the main question — how safely do models respond? — isn't answered yet. Fix: write a clear scoring guide, score all 50 cases, and test more than one model.

Your framing is strong and a great fit for the track: people in care deserts really do lean on chatbots for medical advice, and your four-tier risk taxonomy with regionally grounded scenarios is a sensible way in. The hard part is that the evaluation never actually ran because of API limits, so there are no scores, transcripts, or model comparisons, and your central hypothesis goes untested; there's also no code to inspect. I genuinely appreciate the honesty about why it stalled, but the next version needs data behind it. If you run even the ten Critical-tier scenarios by hand against an explicit rubric and report the real transcripts and scores, you'll give the framework something concrete to stand on, and it's well worth doing.