$β$-calibration of Language Model Confidence Scores for Generative QA
원문 초록
To use generative question-and-answering (QA) systems for decision-making and in any critical application, these systems need to provide well-calibrated confidence scores that reflect the correctness of their answers. Existing calibration methods aim to ensure that the confidence score is on average indicative of the likelihood that the answer is correct. We argue, however, that this standard (average-case) notion of calibration is difficult to interpret for decision-making in generative QA. To address this, we generalize the standard notion of average calibration and introduce $\beta$-calibration, which ensures calibration holds across different question-and-answer groups. We then propose discretized posthoc calibration schemes for achieving $\beta$-calibration.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Inaugural MOASEI Competition at AAMAS'2025: A Technical Report
Case-based Reasoning Augmented Large Language Model Framework for Decision Making in Realistic Safety-Critical Driving Scenarios
Make Your AUV Adaptive: An Environment-Aware Reinforcement Learning Framework For Underwater Tasks
Hypothesis Testing for Quantifying LLM-Human Misalignment in Multiple Choice Settings
CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model
HOP+: History-enhanced and Order-aware Pre-training for Vision-and-Language Navigation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.