Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents
원문 초록
At the heart of improving conversational AI is the open problem of how to evaluate conversations. Issues with automatic metrics are well known (Liu et al., 2016, arXiv:1603.08023), with human evaluations still considered the gold standard. Unfortunately, how to perform human evaluations is also an open problem: differing data collection methods have varying levels of human agreement and statistical sensitivity, resulting in differing amounts of human annotation hours and labor costs. In this work we compare five different crowdworker-based human evaluation methods and find that different methods are best depending on the types of models compared, with no clear winner across the board. While this highlights the open problems in the area, our analysis leads to advice of when to use which one, and possible future directions.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Generative Latent Kernel Modeling for Blind Motion Deblurring
Differentially Private Relational Learning with Entity-level Privacy Guarantees
EtiCor++: Towards Understanding Etiquettical Bias in LLMs
Fragility in Average Treatment Effect on the Treated under Limited Covariate Support
Assessing parameter identifiability of a hemodynamics PDE model using spectral surrogates and dimension reduction
ACCESS DENIED INC: The First Benchmark Environment for Sensitivity Awareness
토론
이 논문과 연결된 포럼 글이 아직 없습니다.