Mismatch between Multi-turn Dialogue and its Evaluation Metric in Dialogue State Tracking
원문 초록
Dialogue state tracking (DST) aims to extract essential information from multi-turn dialogue situations and take appropriate actions. A belief state, one of the core pieces of information, refers to the subject and its specific content, and appears in the form of domain-slot-value. The trained model predicts "accumulated" belief states in every turn, and joint goal accuracy and slot accuracy are mainly used to evaluate the prediction; however, we specify that the current evaluation metrics have a critical limitation when evaluating belief states accumulated as the dialogue proceeds, especially in the most used MultiWOZ dataset. Additionally, we propose relative slot accuracy to complement existing metrics. Relative slot accuracy does not depend on the number of predefined slots, and allows intuitive evaluation by assigning relative scores according to the turn of each dialogue. This study also encourages not solely the reporting of joint goal accuracy, but also various complementary metrics in DST tasks for the sake of a realistic evaluation.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Know Your Mistakes: Towards Preventing Overreliance on Task-Oriented Conversational AI Through Accountability Modeling
Beyond Ontology in Dialogue State Tracking for Goal-Oriented Chatbot
A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understanding
Confidence Estimation for LLM-Based Dialogue State Tracking
Continual Dialogue State Tracking via Reason-of-Select Distillation
TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.