FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows
원문 초록
Despite recent progress in open-domain dialogue evaluation, how to develop automatic metrics remains an open problem. We explore the potential of dialogue evaluation featuring dialog act information, which was hardly explicitly modeled in previous methods. However, defined at the utterance level in general, dialog act is of coarse granularity, as an utterance can contain multiple segments possessing different functions. Hence, we propose segment act, an extension of dialog act from utterance level to segment level, and crowdsource a large-scale dataset for it. To utilize segment act flows, sequences of segment acts, for evaluation, we develop the first consensus-based dialogue evaluation framework, FlowEval. This framework provides a reference-free approach for dialog evaluation by finding pseudo-references. Extensive experiments against strong baselines on three benchmark datasets demonstrate the effectiveness and other desirable characteristics of our FlowEval, pointing out a potential path for better dialogue evaluation.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Chatbots and Dialogue Evaluators
Methods for Recognizing Nested Terms
RuOpinionNE-2024: Extraction of Opinion Tuples from Russian News Texts
Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Models
BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation
Measuring the Robustness of Reference-Free Dialogue Evaluation Systems
토론
이 논문과 연결된 포럼 글이 아직 없습니다.