Continual Pretraining

AG News (AG’s News Corpus) is a subdataset of AG's corpus of news articles constructed by assembling titles and description fields of articles from the 4 largest classes (“World”, “Sports”, “Business”, “Sci/Tech”) of AG’s Corpus. The AG News contains 30,000 training and 1,900 test samples per class. Source: https://arxiv.org/pdf/1509.01626.pdf

SciERC dataset is a collection of 500 scientific abstract annotated with scientific entities, their relations, and coreference clusters. The abstracts are taken from 12 AI conference/workshop proceedings in four AI communities, from the Semantic Scholar Corpus. SciERC extends previous datasets in scientific articles SemEval 2017 Task 10 and SemEval 2018 Task 7 by extending entity types, relation types, relation coverage, and adding cross-sentence relations using coreference links. Source: http://nlp.cs.washington.edu/sciIE/ Image Source: http://nlp.cs.washington.edu/sciIE/

ACL Anthology Reference Corpus (ACL ARC) is a collection of 10,920 academic papers from the ACL Anthology. ACL ARC is cleaned to remove: - files that look like not full papers, paper fragments, foreign-language papers (e.g., French), or pure junk. - headers (title and author information; NOT abstract). - footers ("References" line and the actual references). - some bad characters (spurious characters). - some page numbers (i.e., a single number appearing on a line, with nothing else attached to it). - significant foreign-language (e.g., French) content in an otherwise English paper. The cleaned corpus has 10,628 documents. Source: ACL ARC