LLM Sensitivity Evaluation Framework for Clinical Diagnosis
원문 초록
Large language models (LLMs) have demonstrated impressive performance across various domains. However, for clinical diagnosis, higher expectations are required for LLM's reliability and sensitivity: thinking like physicians and remaining sensitive to key medical information that affects diagnostic reasoning, as subtle variations can lead to different diagnosis results. Yet, existing works focus mainly on investigating the sensitivity of LLMs to irrelevant context and overlook the importance of key information. In this paper, we investigate the sensitivity of LLMs, i.e. GPT-3.5, GPT-4, Gemini, Claude3 and LLaMA2-7b, to key medical information by introducing different perturbation strategies. The evaluation results highlight the limitations of current LLMs in remaining sensitive to key medical information for diagnostic decision-making. The evolution of LLMs must focus on improving their reliability, enhancing their ability to be sensitive to key information, and effectively utilizing this information. These improvements will enhance human trust in LLMs and facilitate their practical application in real-world scenarios. Our code and dataset are available at https://github.com/chenwei23333/DiagnosisQA.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Diffusion-Based Electrocardiography Noise Quantification via Anomaly Detection
Fragility in Average Treatment Effect on the Treated under Limited Covariate Support
Language Agents for Hypothesis-driven Clinical Decision Making with Reinforcement Learning
SPINA Carb: a simple mathematical model supporting fast in-vivo estimation of insulin sensitivity and beta cell function
Dynamic Fusion of Eye Movement Data and Verbal Narrations in Knowledge-rich Domains
Building a Scalable and Interpretable Bayesian Deep Learning Framework for Quality Control of Free Form Surfaces
토론
이 논문과 연결된 포럼 글이 아직 없습니다.