Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks
원문 초록
Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes. Grading these at scale naively, however, is expensive, and rubric-based evaluation has become the dominant scalable alternative. We ask what happens when the rubrics themselves are not airtight, and whether such flaws can be detected and corrected. We apply RIFT, a global rubric failure taxonomy, to three clinical benchmarks (HealthBench Professional, LiveMedBench, and MentalHealthBench), and find failure modes are meaningful: on HealthBench Professional an LLM judge flags 29.6% of criteria as non-atomic and 65.4% as misaligned/rigid, although these rates vary severalfold across five detection judges. Then, we show that these flaws are meaningful and not simply cosmetic. As an example, rewriting bundled criteria of the form "at least one of / all of the following" as equally weighted children and regrading identical responses shifts scores by up to 15.9 percentage points on affected conversations, with disjunctive bundles inflating scores and conjunctive bundles deflating them. Across six answering models and two common graders, these shifts leave the model ranking unchanged, and their net direction depends on how many bundled criteria are penalties. We also find that RIFT can under-detect bundling on clinical rubrics, by an amount that depends on the detection judge: on the same LiveMedBench criteria, judges flag 3.8-15.9% as non-atomic against an estimated 7-19% that are genuinely bundled.
제출 이력
확인된 버전만 표시합니다. 전체 제출 이력은 arXiv 원문에서 확인하세요.
- arXiv:2609.16023v2제출 시각 미확인
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
공통 태그로 연결된 공개 논문이 아직 없습니다.
토론
이 논문과 연결된 포럼 글이 아직 없습니다.