데이터셋 탐색
데이터셋의 설명·홈페이지를 찾고, 관련 태스크의 평가 기록을 살펴보세요.
WikiText-2
WikiText-2 설명 보기
The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License. Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over 110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation and numbers - all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models that can take advantage of long term dependencies. Source: The WikiText Long Term Dependency Language Modeling Dataset Image Source: https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/
WikiText-103
WikiText-103 설명 보기
The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License. Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over 110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation and numbers - all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models that can take advantage of long term dependencies. Source: The WikiText Long Term Dependency Language Modeling Dataset Image Source: https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/
WikiSQL
WikiSQL 설명 보기
WikiSQL consists of a corpus of 87,726 hand-annotated SQL query and natural language question pairs. These SQL queries are further split into training (61,297 examples), development (9,145 examples) and test sets (17,284 examples). It can be used for natural language inference tasks related to relational databases. Source: SQL-to-Text Generation with Graph-to-Sequence Model Image Source: https://blog.einstein.ai/how-to-talk-to-your-database/
Wikipedia Squirrel
Wiki Squirrel 설명 보기
The data was collected from the English Wikipedia (December 2018). These datasets represent page-page networks on specific topics (chameleons, crocodiles and squirrels). Nodes represent articles and edges are mutual links between them. The edges csv files contain the edges - nodes are indexed from 0. The features json files contain the features of articles - each key is a page id, and node features are given as lists. The presence of a feature in the feature list means that an informative noun appeared in the text of the Wikipedia article. The target csv contains the node identifiers and the average monthly traffic between October 2017 and November 2018 for each page. For each page-page network we listed the number of nodes an edges with some other descriptive statistics.
Wikipedia open-domain Question Answering
WikiQA 설명 보기
The WikiQA corpus is a publicly available set of question and sentence pairs, collected and annotated for research on open-domain question answering. In order to reflect the true information need of general users, Bing query logs were used as the question source. Each question is linked to a Wikipedia page that potentially has the answer. Because the summary section of a Wikipedia page provides the basic and usually most important information about the topic, sentences in this section were used as the candidate answers. The corpus includes 3,047 questions and 29,258 sentences, where 1,473 sentences were labeled as answer sentences to their corresponding questions. Source: http://aka.ms/WikiQA Image Source: Yang et al
Wizard of Wikipedia
Wizard of Wikipedia 설명 보기
Wizard of Wikipedia is a large dataset with conversations directly grounded with knowledge retrieved from Wikipedia. It is used to train and evaluate dialogue systems for knowledgeable open dialogue with clear grounding
WikiHow
WikiHow 설명 보기
WikiHow is a dataset of more than 230,000 article and summary pairs extracted and constructed from an online knowledge base written by different human authors. The articles span a wide range of topics and represent high diversity styles. Source: WikiHow: A Large Scale Text Summarization Dataset Image Source: WikiHow: A Large Scale Text Summarization Dataset
Wiki-CS
Wiki-CS 설명 보기
Wiki-CS is a Wikipedia-based dataset for benchmarking Graph Neural Networks. The dataset is constructed from Wikipedia categories, specifically 10 classes corresponding to branches of computer science, with very high connectivity. The node features are derived from the text of the corresponding articles. They were calculated as the average of pretrained GloVe word embeddings (Pennington et al., 2014), resulting in 300-dimensional node features. The dataset has 11,701 nodes and 216,123 edges. Source: Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks Image Source: https://arxiv.org/pdf/2007.02901v1.pdf
2WikiMultiHopQA
2WikiMultiHopQA 설명 보기
Uses structured and unstructured data. The dataset introduces the evidence information containing a reasoning path for multi-hop questions. Source: Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
WikiArt
WikiArt 설명 보기
WikiArt contains painting from 195 different artists. The dataset has 42129 images for training and 10628 images for testing. Source: Adding New Tasks to a Single Network with Weight Transformations using Binary Masks Image Source: https://towardsdatascience.com/the-non-treachery-of-dataset-df1f6cbe577e
WikiMatrix
WikiMatrix 설명 보기
WikiMatrix is a dataset of parallel sentences in the textual content of Wikipedia for all possible language pairs. The mined data consists of: - 85 different languages, 1620 language pairs - 134M parallel sentences, out of which 34M are aligned with English Source: WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia
Wikipedia Biography Dataset
WikiBio 설명 보기
This dataset gathers 728,321 biographies from English Wikipedia. It aims at evaluating text generation algorithms. For each article, we provide the first paragraph and the infobox (both tokenized).
Wikipedia-based Image Text
WIT 설명 보기
Wikipedia-based Image Text (WIT) Dataset is a large multimodal multilingual dataset. WIT is composed of a curated set of 37.6 million entity rich image-text examples with 11.5 million unique images across 108 Wikipedia languages. Its size enables WIT to be used as a pretraining dataset for multimodal machine learning models. Key Advantages A few unique advantages of WIT: - The largest multimodal dataset (time of this writing) by the number of image-text examples. - A massively multilingual (first of its kind) with coverage for over 100+ languages. - A collection of diverse set of concepts and real world entities. - Brings forth challenging real-world test sets.
WikiTableQuestions
WikiTableQuestions 설명 보기
WikiTableQuestions is a question answering dataset over semi-structured tables. It is comprised of question-answer pairs on HTML tables, and was constructed by selecting data tables from Wikipedia that contained at least 8 rows and 5 columns. Amazon Mechanical Turk workers were then tasked with writing trivia questions about each table. WikiTableQuestions contains 22,033 questions. The questions were not designed by predefined templates but were hand crafted by users, demonstrating high linguistic variance. Compared to previous datasets on knowledge bases it covers nearly 4,000 unique column headers, containing far more relations than closed domain datasets and datasets for querying knowledge bases. Its questions cover a wide range of domains, requiring operations such as table lookup, aggregation, superlatives (argmax, argmin), arithmetic operations, joins and unions. Source: Explaining Queries over Web Tables to Non-Experts Image Source: https://ppasupat.github.io/WikiTableQuestions/
IMDB-WIKI
IMDB-WIKI 설명 보기
To the best of our knowledge this is the largest publicly available dataset of face images with gender and age labels for training. We provide pretrained models for both age and gender prediction.
PAN-X
WikiANN 설명 보기
WikiANN, also known as PAN-X, is a multilingual named entity recognition dataset. It consists of Wikipedia articles that have been annotated with LOC (location), PER (person), and ORG (organization) tags in the IOB2 format¹². This dataset serves as a valuable resource for training and evaluating named entity recognition models across various languages. For instance, it includes information about notable individuals, places, and organizations mentioned in Wikipedia articles. Researchers and practitioners can use WikiANN to develop and improve natural language processing systems that identify and classify named entities in text. (1) wikiann · Datasets at Hugging Face. https://huggingface.co/datasets/wikiann. (2) wikiann | TensorFlow Datasets. https://tensorflow.google.cn/datasets/catalog/wikiann. (3) wikiann · Datasets at Hugging Face. https://huggingface.co/datasets/wikiann/viewer/en. (4) WikiAnn Dataset | Papers With Code. https://paperswithcode.com/dataset/wikiann-1.
WikiLarge
WikiLarge 설명 보기
WikiLarge comprise 359 test sentences, 2000 development sentences and 300k training sentences. Each source sentences in test set has 8 simplified references Source: Semi-Supervised Text Simplification with Back-Translation and Asymmetric Denoising Autoencoders Image Source: https://arxiv.org/pdf/1904.02767.pdf
WikiHop
WikiHop 설명 보기
WikiHop is a multi-hop question-answering dataset. The query of WikiHop is constructed with entities and relations from WikiData, while supporting documents are from WikiReading. A bipartite graph connecting entities and documents is first built and the answer for each query is located by traversal on this graph. Candidates that are type-consistent with the answer and share the same relation in query with the answer are included, resulting in a set of candidates. Thus, WikiHop is a multi-choice style reading comprehension data set. There are totally about 43K samples in training set, 5K samples in development set and 2.5K samples in test set. The test set is not provided. The task is to predict the correct answer given a query and multiple supporting documents. The dataset includes a masked variant, where all candidates and their mentions in the supporting documents are replaced by random but consistent placeholder tokens. Source: Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs Image Source: http://qangaroo.cs.ucl.ac.uk/
WikiSum
WikiSum 설명 보기
WikiSum is a dataset based on English Wikipedia and suitable for a task of multi-document abstractive summarization. In each instance, the input is comprised of a Wikipedia topic (title of article) and a collection of non-Wikipedia reference documents, and the target is the Wikipedia article text. The dataset is restricted to the articles with at least one crawlable citation. The official split divides the articles roughly into 80/10/10 for train/development/test subsets, resulting in 1865750, 233252, and 232998 examples respectively. Source: Generating Wikipedia by Summarizing Long Sequences Image Source: https://arxiv.org/pdf/1801.10198.pdf
Wikidata5M
Wikidata5M 설명 보기
Wikidata5m is a million-scale knowledge graph dataset with aligned corpus. This dataset integrates the Wikidata knowledge graph and Wikipedia pages. Each entity in Wikidata5m is described by a corresponding Wikipedia page, which enables the evaluation of link prediction over unseen entities. The dataset is distributed as a knowledge graph, a corpus, and aliases. We provide both transductive and inductive data splits used in the original paper.
WikiLingua
WikiLingua 설명 보기
WikiLingua includes ~770k article and summary pairs in 18 languages from WikiHow. Gold-standard article-summary alignments across languages are extracted by aligning the images that are used to describe each how-to step in an article. Source: https://github.com/esdurmus/Wikilingua Image Source: Ladhak et al
WikiMovies
WikiMovies 설명 보기
WikiMovies is a dataset for question answering for movies content. It contains ~100k questions in the movie domain, and was designed to be answerable by using either a perfect KB (based on OMDb), Source: WikiMovies
WikiEvents
WikiEvents 설명 보기
WikiEvents is a document-level event extraction benchmark dataset which includes complete event and coreference annotation.
Wikipedia Current Events Portal
WCEP 설명 보기
The WCEP dataset for multi-document summarization (MDS) consists of short, human-written summaries about news events, obtained from the Wikipedia Current Events Portal (WCEP), each paired with a cluster of news articles associated with an event. These articles consist of sources cited by editors on WCEP, and are extended with articles automatically obtained from the Common Crawl News dataset. Source: WCEP
Wiki-40B
Wiki-40B 설명 보기
A new multilingual language model benchmark that is composed of 40+ languages spanning several scripts and linguistic families containing round 40 billion characters and aimed to accelerate the research of multilingual modeling. Source: Wiki-40B: Multilingual Language Model Dataset
WikiCoref
WikiCoref 설명 보기
WikiCoref is an English corpus annotated for anaphoric relations, where all documents are from the English version of Wikipedia. Source: WikiCoref: An English Coreference-annotated Corpus of Wikipedia Articles
WikiReading
WikiReading 설명 보기
WikiReading is a large-scale natural language understanding task and publicly-available dataset with 18 million instances. The task is to predict textual values from the structured knowledge base Wikidata by reading the text of the corresponding Wikipedia articles. The task contains a rich variety of challenging classification and extraction sub-tasks, making it well-suited for end-to-end models such as deep neural networks (DNNs). Source: WIKIREADING: A Novel Large-scale Language Understanding Task over Wikipedia Image Source: Hewlett et al
Wiki-ZSL
Wiki-ZSL 설명 보기
The Wiki-ZSL (Wiki Zero-Shot Learning) dataset contains 113 relations and 94,383 instances from Wikipedia. The dataset is divided into three subsets: training set (98 relations), validation set (5 relations) and test set (10 relations).
WikiSplit
WikiSplit 설명 보기
Contains one million naturally occurring sentence rewrites, providing sixty times more distinct split examples and a ninety times larger vocabulary than the WebSplit corpus introduced by Narayan et al. (2017) as a benchmark for this task. Source: Learning To Split and Rephrase From Wikipedia Edit History
WikiSection
WikiSection 설명 보기
A publicly available dataset with 242k labeled sections in English and German from two distinct domains: diseases and cities. Source: SECTOR: A Neural Model for Coherent Topic Segmentation and Classification
Wiki-One
Wiki-One 설명 보기
This dataset is a Wikipedia dump, split by relations to perform Few-Shot Knowledge Graph Completion. \begin{table}[] \begin{tabular}{@{}lllccl@{}} \textbf{Dataset} & \textbf{# Ent} & \textbf{# Rel} & \textbf{# Triplets} & \textbf{Train/Dev/Test} \ Wiki-One & 4,838,244 & 822 & 5,829,240 & 133/16/34 \ \end{tabular} \caption{Datasets used in the experiments. } \end{table}
GenWiki
GenWiki 설명 보기
GenWiki is a large-scale dataset for knowledge graph-to-text (G2T) and text-to-knowledge graph (T2G) conversion. It is introduced in the paper "GenWiki: A Dataset of 1.3 Million Content-Sharing Text and Graphs for Unsupervised Graph-to-Text Generation" by Zhijing Jin, Qipeng Guo, Xipeng Qiu, and Zheng Zhang at COLING 2020.
WikiAtomicEdits
WikiAtomicEdits 설명 보기
WikiAtomicEdits is a corpus of 43 million atomic edits across 8 languages. These edits are mined from Wikipedia edit history and consist of instances in which a human editor has inserted a single contiguous phrase into, or deleted a single contiguous phrase from, an existing sentence. Source: WikiAtomicEdits: A Multilingual Corpus of Wikipedia Edits for Modeling Language and Discourse Image Source: https://arxiv.org/pdf/1808.09422v1.pdf
WikiConv
WikiConv 설명 보기
A corpus that encompasses the complete history of conversations between contributors to Wikipedia, one of the largest online collaborative communities. By recording the intermediate states of conversations---including not only comments and replies, but also their modifications, deletions and restorations---this data offers an unprecedented view of online conversation. Source: WikiConv: A Corpus of the Complete Conversational History of a Large Online Collaborative Community
WikiTableT
WikiTableT 설명 보기
WikiTableT contains Wikipedia article sections and their corresponding tabular data and various metadata. WikiTableT contains millions of instances while covering a broad range of topics and a variety of kinds of generation tasks. Source: WikiTableT
WikiCatSum
WikiCatSum 설명 보기
WikiCatSum is a domain specific Multi-Document Summarisation (MDS) dataset. It assumes the summarisation task of generating Wikipedia lead sections for Wikipedia entities of a certain domain (e.g. Companies) from the set of documents cited in Wikipedia articles or returned by Google (using article titles as queries). The dataset includes three domains: Companies, Films, and Animals. Source: https://datashare.ed.ac.uk/handle/10283/3368
Wikipedia Detox
WikiDetox 설명 보기
An annotated dataset of 1m crowd-sourced annotations that cover 100k talk page diffs (with 10 judgements per diff) for personal attacks, aggression, and toxicity.
WikiTables-TURL
WikiTables-TURL 설명 보기
The WikiTables-TURL dataset was constructed by the authors of TURL and is based on the WikiTable corpus, which is a large collection of Wikipedia tables. The dataset consists of 580,171 tables divided into fixed training, validation and testing splits. Additionally, the dataset contains metadata about each table, such as the table name, table caption and column headers. 406,706 of these tables are annotated for the Column Type Annotation (CTA) task, 55,970 tables for the Columns Property Annotation (CPA) task and 200,744 tables for the Cell Entity Annotation (CEA) task. As classes for the CTA and CPA, Freebase's types and relations were used, whereas for the CEA task entities from Freebase were used. The table below lists the total annotated columns (or cells in the case of CEA) for each split and for each task as well as the number of classes used for annotation. | | Training | Validation | Testing | Classes | |-----|--------|----------|-------|-------| | CTA | 628,254 |13,391| 13,025 | 255 | | CPA | 62,954| 2,175 | 2,072 | 121 | | CEA | 1,264,217 | 76,720 | 225,777 | 1,787,737 | The authors have made the dataset and its variants publicly available for download.
French Wikipedia
French Wikipedia 설명 보기
French Wikipedia is a dataset used for pretraining the CamemBERT French language model. It uses the official 2019 French Wikipedia dumps
SimpleQuestionsWikiData
SimpleQuestionsWikiData 설명 보기
SimpleQuestionsWikidata maps SimpleQuestions to Wikidata. It was proposed in the paper Question Answering Benchmarks for Wikidata by Diefenbach et al.
WikiCREM
WikiCREM 설명 보기
An unsupervised dataset for co-reference resolution. Presented in the publication: Kocijan et. al, WikiCREM: A Large Unsupervised Corpus for Coreference Resolution, presented at EMNLP 2019. Source: WikiCREM Image Source: https://arxiv.org/pdf/1908.08025v3.pdf
WikiNEuRal
WikiNEuRal 설명 보기
WikiNEuRal is a high-quality automatically-generated dataset for Multilingual Named Entity Recognition.
WikiNews Arabic Diacritization Benchmark Dataset
WikiNews Dataset 설명 보기
The WikiNews Arabic Diacritization dataset is a test set composed of 70 WikiNews articles (majority are from 2013 and 2014) that cover a variety of themes, namely: politics, economics, health, science and technology, sports, arts, and culture. The articles are evenly distributed among the different themes (10 per theme). The articles contain 18,300 words with around 400 different sentences (Each line is considered as a sentence).
WikiHowQA
WikiHowQA 설명 보기
WikiHowQA is a Community-based Question Answering dataset, which can be used for both answer selection and abstractive summarization tasks. It contains 76,687 questions in the train set, 8,000 in the development set and 22,354 in the test set. Source: WikiHowQA
Wikipedia Person and Animal Dataset
Wikipedia Person and Animal Dataset 설명 보기
This dataset gathers 428,748 person and 12,236 animal infobox with descriptions based on Wikipedia dump (2018/04/01) and Wikidata (2018/04/12).
GMEG-wiki
GMEG-wiki 설명 보기
Grammatical error correction dataset for text from Wikipedia.
InferWiki
InferWiki 설명 보기
InferWiki is a Knowledge Graph Completion (KGC) dataset that improves upon existing benchmarks in inferential ability, assumptions, and patterns. First, each testing sample is predictable with supportive data in the training set. Second, InferWiki initiates the evaluation following the open-world assumption and improves the inferential difficulty of the closed-world assumption, by providing manually annotated negative and unknown triples. Third, the dataset includes various inference patterns (e.g., reasoning path length and types) for comprehensive evaluation.
TemporalWiki
TemporalWiki 설명 보기
TemporalWiki is a lifelong benchmark for ever-evolving LMs that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation, respectively. The benchmark hence allows researchers to periodically track an LM's ability to retain previous knowledge and acquire updated/new knowledge at each point in time.
WIKIPerson
WIKIPerson 설명 보기
WIKIPerson is a high-quality human-annotated visual person linking dataset based on Wikipedia. The dataset contains a total of 48k different news images, covering 13k out of 120K Person named entities, each of which corresponds to a celebrity in Wikipedia. Unlike previously commonly-used datasets in EL, the mention in WIKIPerson is only an image containing the person entity with its bounding box. The corresponding label identifies a unique entity in Wikipedia. For each entity in the Wikipedia, we provide textual descriptions as well as images to satisfy the need of three sub-tasks. Source: Visual Named Entity Linking: A New Dataset and A Baseline Image Source: https://arxiv.org/pdf/2211.04872v1.pdf