Evaluating Clinical Competencies of Large Language Models with a General Practice Benchmark
원문 초록
Large Language Models (LLMs) have demonstrated considerable potential in general practice. However, existing benchmarks and evaluation frameworks primarily depend on exam-style or simplified question-answer formats, lacking a competency-based structure aligned with the real-world clinical responsibilities encountered in general practice. Consequently, the extent to which LLMs can reliably fulfill the duties of general practitioners (GPs) remains uncertain. In this work, we propose a novel evaluation framework to assess the capability of LLMs to function as GPs. Based on this framework, we introduce a general practice benchmark (GPBench), whose data are meticulously annotated by domain experts in accordance with routine clinical practice standards. We evaluate ten state-of-the-art LLMs and analyze their competencies. Our findings indicate that current LLMs are not yet ready for deployment in such settings without human oversight, and further optimization specifically tailored to the daily responsibilities of GPs is essential.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Training-free LLM Merging for Multi-task Learning
STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean
Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Models
Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.