Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics
원문 초록
In text summarization, evaluating the efficacy of automatic metrics without human judgments has become recently popular. One exemplar work concludes that automatic metrics strongly disagree when ranking high-scoring summaries. In this paper, we revisit their experiments and find that their observations stem from the fact that metrics disagree in ranking summaries from any narrow scoring range. We hypothesize that this may be because summaries are similar to each other in a narrow scoring range and are thus, difficult to rank. Apart from the width of the scoring range of summaries, we analyze three other properties that impact inter-metric agreement - Ease of Summarization, Abstractiveness, and Coverage. To encourage reproducible research, we make all our analysis code and data publicly available.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
APE: Selective Fine-tuning with Acceptance Criteria for Language Model Adaptation
On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention
Improving large language models with concept-aware fine-tuning
Multimodal Survival Modeling in the Age of Foundation Models
A Split-then-Join Approach to Abstractive Summarization for Very Long Documents in a Low Resource Setting
ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphs
토론
이 논문과 연결된 포럼 글이 아직 없습니다.