Summarization

The PubMed dataset consists of 19717 scientific publications from PubMed database pertaining to diabetes classified into one of three classes. The citation network consists of 44338 links. Each publication in the dataset is described by a TF/IDF weighted word vector from a dictionary which consists of 500 unique words. Source: https://linqs.soe.ucsc.edu/data

CNN/Daily Mail is a dataset for text summarization. Human generated abstractive summary bullets were generated from news stories in CNN and Daily Mail websites as questions (with one of the entities hidden), and stories as the corresponding passages from which the system is expected to answer the fill-in the-blank question. The authors released the scripts that crawl, extract and generate pairs of passages and questions from these websites. In all, the corpus has 286,817 training pairs, 13,368 validation pairs and 11,487 test pairs, as defined by their scripts. The source documents in the training set have 766 words spanning 29.74 sentences on an average while the summaries consist of 53 words and 3.72 sentences. Source: Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond

A new dataset with abstractive dialogue summaries. Source: SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization

Multi-News, consists of news articles and human-written summaries of these articles from the site newser.com. Each summary is professionally written by editors and includes links to the original articles cited. Source: Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model Image Source: https://arxiv.org/pdf/1906.01749.pdf

Consists of 1.3 million records of U.S. patent documents along with human written abstractive summaries. Source: BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization

BillSum is the first dataset for summarization of US Congressional and California state bills. The BillSum dataset consists of three parts: US training bills, US test bills and California test bills. The US bills were collected from the Govinfo service provided by the United States Government Publishing Office (GPO). The corpus consists of bills from the 103rd-115th (1993-2018) sessions of Congress. The data was split into 18,949 train bills and 3,269 test bills. For California, bills from the 2015-2016 session were scraped directly from the legislature’s website; the summaries were written by their Legislative Counsel. The BillSum corpus focuses on mid-length legislation from 5,000 to 20,000 character in length. The authors chose to measure the text length in characters, instead of words or sentences, because the texts have complex structure that makes it difficult to consistently measure words. The range was chosen because on one side, short bills introduce minor changes and do not require summaries. While the CRS produces summaries for them, they often contain most of the text of the bill. On the other side, very long legislation is often composed of several large sections. Source: BillSum: A Corpus for Automatic Summarization of US Legislation

A large-scale MultiLingual SUMmarization dataset. Obtained from online newspapers, it contains 1.5M+ article/summary pairs in five different languages -- namely, French, German, Spanish, Russian, Turkish. Together with English newspapers from the popular CNN/Daily mail dataset, the collected data form a large scale multilingual dataset which can enable new research directions for the text summarization community. Source: MLSUM: The Multilingual Summarization Corpus

The Extreme Summarization (XSum) dataset is a dataset for evaluation of abstractive single-document summarization systems. The goal is to create a short, one-sentence new summary answering the question “What is the article about?”. The dataset consists of 226,711 news articles accompanied with a one-sentence summary. The articles are collected from BBC articles (2010 to 2017) and cover a wide variety of domains (e.g., News, Politics, Sports, Weather, Business, Technology, Science, Health, Family, Education, Entertainment and Arts). The official random split contains 204,045 (90%), 11,332 (5%) and 11,334 (5) documents in training, validation and test sets, respectively. Source: https://arxiv.org/pdf/1808.08745.pdf Image Source: https://arxiv.org/pdf/1808.08745.pdf

This is a dataset for evaluating summarisation methods for research papers. Source: A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents

FacetSum is a faceted summarization dataset for scientific documents. FacetSum has been built on Emerald journal articles, covering a diverse range of domains. Different from traditional document-summary pairs, FacetSum provides multiple summaries, each targeted at specific sections of a long document, including the purpose, method, findings, and value.

MuLD (Multitask Long Document Benchmark) is a set of 6 NLP tasks where the inputs consist of at least 10,000 words. The benchmark covers a wide variety of task types including translation, summarization, question answering, and classification. Additionally there is a range of output lengths from a single word classification label all the way up to an output longer than the input text.

The dataset contains 140 paragraphs from climate change reports with associated aspect-based (i.e. query-focused) summaries, that were produced by experts especially for policy-makers. The dataset's format is the one required by Hugginface's datasets library () and it can be loaded by looking at the available examples. The dataset consists of three possible configurations: "AR6" to download just the latest climate change report (70 paragraph/summaries), "AR5" to download just the first climate change report (70 paragraph/summaries) or "ALL" to download both (140 paragraph/summaries). The available features in the dataset are: "full_paragraphs": Full paragraphs to summarise, "summary": The current query-focused ground truth summary, "summary_topic": The topic of on which the current summary is focusing on, "paragraph_topic": The macro-topic of the paragraph(s), "section_topic": The high-level topic of the section from which the current summary originate, "source": The source document (AR5/AR6), "paragraph_ids": The id(s) of the current paragraph(s), "paragraph_titles": The titles of each of the summarised paragraphs, "ID": The row ID All values above are strings. Please notice that each ground truth summary might refer to one or more target paragraphs from the target document, as the summaries transverse the reports according to specific topics (which is indicated in each case by the summary_topic variable). To better understand the format of this dataset the original IPCC reports and the accompanying summary for policymakers from which the ground truth summaries are taken can be consulted at: https://www.ipcc.ch/

We provide a new data set XWikiRef for the task of Cross-lingual Multi-document Summarization. This task aims at generating Wikipedia style text in Low Resource languages by taking reference text as input. Overall, the data set contains 8 different languages: bengali (bn), english (en), hindi (hi), marathi (mr), malayalam (ml), odia (or), punjabi (pa) and tamil (ta). It also contains 5 domains: books, films, politicians, sportsman and writers. ##Data Format Dataset is publicly available here. Each directory contains language specific data subset having 1 json file per domain. In each file, each line denotes one article. It contains the following set of keys: - Article title - Sections - section title 1 - section text 1 - list of reference texts 1 - ..... - ..... - ..... - section title n - section text n - list of reference texts 1