News Summarization and Evaluation in the Era of GPT-3
원문 초록
The recent success of prompting large language models like GPT-3 has led to a paradigm shift in NLP research. In this paper, we study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, prompted using only a task description, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics cannot reliably evaluate GPT-3 summaries. Finally, we evaluate models on a setting beyond generic summarization, specifically keyword-based summarization, and show how dominant fine-tuning approaches compare to prompting. To support further research, we release: (a) a corpus of 10K generated summaries from fine-tuned and prompt-based models across 4 standard summarization benchmarks, (b) 1K human preference judgments comparing different systems for generic- and keyword-based summarization.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
APE: Selective Fine-tuning with Acceptance Criteria for Language Model Adaptation
Unraveling the Capabilities of Language Models in News Summarization
German also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Dataset
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization
GreekT5: A Series of Greek Sequence-to-Sequence Models for News Summarization
Explainable News Summarization -- Analysis and mitigation of Disagreement Problem
토론
이 논문과 연결된 포럼 글이 아직 없습니다.