Definite Pronoun Resolution Dataset
Composes sentence pairs (i.e., twin sentences). Source: Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge
태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.
Natural Language Understanding is an important field of Natural Language Processing which contains various tasks such as text classification, natural language inference and story comprehension. Applications enabled by natural language understanding range from question answering to automated reasoning.
Source: Find a Reasonable Ending for Stories: Does Logic Relation Help the Story Cloze Test?
출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.
Composes sentence pairs (i.e., twin sentences). Source: Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge
The Medical Dataset for Abbreviation Disambiguation for Natural Language Understanding (MeDAL) is a large medical text dataset curated for abbreviation disambiguation, designed for natural language understanding pre-training in the medical domain. It was published at the ClinicalNLP workshop at EMNLP. Source: https://github.com/McGill-NLP/medal Image Source: https://github.com/McGill-NLP/medal
An IMPlicature and PRESupposition diagnostic dataset (IMPPRES), consisting of >25k semiautomatically generated sentence pairs illustrating well-studied pragmatic inference types. Source: Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition
nlu++ is a dataset for natural language understanding (NLU) in task-oriented dialogue (ToD) systems, with the aim to provide a much more challenging evaluation environment for dialogue NLU models, up to date with the current application and industry requirements. nlu++ is divided into two domains (banking and hotels) and brings several crucial improvements over current commonly used NLU datasets. 1) Nlu++ provides fine-grained domain ontologies with a large set of challenging multi-intent sentences, introducing and validating the idea of intent modules that can be combined into complex intents that convey complex user goals, combined with finer-grained and thus more challenging slot sets. 2) The ontology is divided into domain-specific and generic (i.e., domain-universal) intent modules that overlap across domains, promoting cross-domain reusability of annotated examples. 3) The dataset design has been inspired by the problems observed in industrial ToD systems, and 4) it has been collected, filtered and carefully annotated by dialogue NLU experts, yielding high-quality annotated data. List of datasets: Banking: online banking queries annotated with their corresponding intents. Span Extraction: the data used for the SpanConvert paper. NLU++: a challenging evaluation environment for dialogue NLU models (multi-domain, multi-label intents and slots). EVI: a challenging multilingual dataset for knowledge-based enrollment, identification, and identification in spoken dialogue systems.
VisPro dataset contains coreference annotation of 29,722 pronouns from 5,000 dialogues. Source: VisPro
An unsupervised dataset for co-reference resolution. Presented in the publication: Kocijan et. al, WikiCREM: A Large Unsupervised Corpus for Coreference Resolution, presented at EMNLP 2019. Source: WikiCREM Image Source: https://arxiv.org/pdf/1908.08025v3.pdf
A large-scale evaluation set that provides human ratings for the plausibility of 10,000 SP pairs over five SP relations, covering 2,500 most frequent verbs, nouns, and adjectives in American English. Source: SP-10K: A Large-scale Evaluation Set for Selectional Preference Acquisition
XWINO is a multilingual collection of Winograd Schemas in six languages that can be used for evaluation of cross-lingual commonsense reasoning capabilities. The datasets that comprise XWINO are: * Source: The original Winograd Schema Challenge (Levesque, 2012); * Source: Additional data from the SuperGLUE WSC benchmark (Wang et al., 2019); * Source: The Definite Pronoun Resolution dataset (Rahman and Ng, 2012) (accessed from https://github.com/Yre/wsc_naive); * Source: A collection of French Winograd Schemas (Amsili and Seminck, 2017); * Source: Japanese translation of Winograd Schema Challenge (柴田知秀 et al., 2015); * Source: Russian Winograd Schema Challenge (Shavrina et al., 2020); * Source: A collection of Winograd Schemas in Chinese; * Source: Winograd Schemas in Portuguese (Melo et al., 2019).
A comprehensive multi-task benchmark for the Polish language understanding, accompanied by an online leaderboard. It consists of a diverse set of tasks, adopted from existing datasets for named entity recognition, question-answering, textual entailment, and others. Source: KLEJ: Comprehensive Benchmark for Polish Language Understanding
A benchmark Arabic dataset for commonsense understanding and validation as well as a baseline research and models trained using the same dataset. Source: Is this sentence valid? An Arabic Dataset for Commonsense Validation
Contains 1507 domain-expert annotated consumer health questions and corresponding summaries. The dataset is derived from the community question answering forum and therefore provides a valuable resource for understanding consumer health-related posts on social media.
Emotional Dialogue Acts data contains dialogue act labels for existing emotion multi-modal conversational datasets. We chose two popular multimodal emotion datasets: Multimodal EmotionLines Dataset (MELD) and Interactive Emotional dyadic MOtion CAPture database (IEMOCAP). EDAs reveal associations between dialogue acts and emotional states in a natural-conversational language such as Accept/Agree dialogue acts often occur with the Joy emotion, Apology with Sadness, and Thanking with Joy.
A massive, deduplicated corpus of 7.4M Python files from GitHub. Source: Learning and Evaluating Contextual Embedding of Source Code
GeoGLUE is a GeoGraphic Language Understanding Evaluation benchmark, which consists of six geographic text-related tasks, including geographic textual similarity on recall, geotagged geographic elements tagging, geographic composition analysis, geographic where what cut, and geographic entity alignment. All tasks' datasets are collected from open-released resources. Source: GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmark Image Source: GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmark
The dataset contains 3304 cases from the Supreme Court of the United States from 1955 to 2021. Each case has the case's identifiers as well as the facts of the case and the decision outcome. Other related datasets rarely included the facts of the case which could prove to be helpful in natural language processing applications. One potential use case of this dataset is determining the outcome of a case using its facts.
MuLD (Multitask Long Document Benchmark) is a set of 6 NLP tasks where the inputs consist of at least 10,000 words. The benchmark covers a wide variety of task types including translation, summarization, question answering, and classification. Additionally there is a range of output lengths from a single word classification label all the way up to an output longer than the input text.
NusaCrowd is a collaborative initiative to collect and unite existing resources for Indonesian languages, including opening access to previously non-public resources. Through this initiative, the authors have has brought together 137 datasets and 117 standardized data loaders. The quality of the datasets has been assessed manually and automatically, and their effectiveness has been demonstrated in multiple experiments. Source: https://arxiv.org/pdf/2212.09648v2.pdf Image Source: NusaCrowd: Open Source Initiative for Indonesian NLP Resources
WikiText-TL-39 is a benchmark language modeling dataset in Filipino that has 39 million tokens in the training set. Source: Evaluating Language Model Finetuning Techniques for Low-resource Languages
RuMedBench is a benchmark dataset for Russian medical language understanding.
This datasets consists of challenging reasoning questions in multiple choice format.
The CANDOR corpus is a large, novel, multimodal corpus of 1,656 recorded conversations in spoken English. This 7+ million word, 850 hour corpus totals over 1TB of audio, video, and transcripts, with moment-to-moment measures of vocal, facial, and semantic expression, along with an extensive survey of speaker post conversation reflections.
ChatLog is a coarse-to-fine temporal dataset called ChatLog, consisting of two parts that update monthly and daily: 1. ChatLog-Monthly is a dataset of 38,730 question-answer pairs collected every month including questions from both the reasoning and classification tasks. 2. ChatLogDaily, on the other hand, consists of ChatGPT's responses to 1000 identical questions for long-form generation every day. Source: ChatLog: Recording and Analyzing ChatGPT Across Time Image Source: ChatLog | Github Repo
CLUES (Constrained Language Understanding Evaluation Standard) is a benchmark for evaluating the few-shot learning capabilities of NLU models.
Dialog-based Language Learning dataset is designed to measure how well models can perform at learning as a student given a teacher’s textual responses to the student’s answer (as well as potentially receiving an external real-valued reward signal). Source: Dialog-based Language Learning dataset
Dataset Summary The DiscoEval is an English-language Benchmark that contains a test suite of 7 tasks to evaluate whether sentence representations include semantic information relevant to discourse processing. The benchmark datasets offer a collection of tasks designed to evaluate natural language understanding models in the context of discourse analysis and coherence. Dataset Sources Arxiv: A repository of scientific papers and research articles. Wikipedia: An extensive online encyclopedia with articles on diverse topics. Rocstory: A dataset consisting of fictional stories. Ubuntu IRC channel: Conversational data extracted from the Ubuntu Internet Relay Chat (IRC) channel. PeerRead: A dataset of scientific papers frequently used for discourse-related tasks. RST Discourse Treebank: A dataset annotated with Rhetorical Structure Theory (RST) discourse relations. Penn Discourse Treebank: Another dataset with annotated discourse relations, facilitating the study of discourse structure
EPIC30M contains a subset of 26.2 millions tweets related to three general diseases, namely Ebola, Cholera and Swine Flu, and another subset of 4.7 millions tweets of six global epidemic outbreaks, including 2009 H1N1 Swine Flu, 2010 Haiti Cholera, 2012 Middle-East Respiratory Syndrome (MERS), 2013 West African Ebola, 2016 Yemen Cholera and 2018 Kivu Ebola. Source: EPIC30M: An Epidemics Corpus Of Over 30 Million Relevant Tweets
ExPUNations is a humor dataset with such extensive and fine-grained annotations specifically for puns. This dataset is designed for two new tasks namely, explanation generation to aid with pun classification and keyword-conditioned pun generation Source: ExPUNations: Augmenting Puns with Keywords and Explanations Image Source: https://arxiv.org/pdf/2210.13513v1.pdf
IDK-MRC is an Indonesian Machine Reading Comprehension (MRC) dataset consists of more than 10K questions in total with over 5K unanswerable questions with diverse question types. Source: IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension Image Source: https://arxiv.org/pdf/2210.13778v1.pdf
ImagiFilter focusses on photographic and/or natural images, a very common use-case in computer vision research. Annotations for coarse prediction are provided, i.e. photographic vs. non-photographic, and smaller fine-grained prediction tasks where the non-photographic class is broken down into five classes: maps, drawings, graphs, icons, and sketches. Source: ImagiFilter: A resource to enable the semi-automatic mining of images at scale
MAVEN-Arg is an advanced event argument extraction dataset, which offers three main advantages: * A comprehensive schema covering 162 event types and 612 argument roles, all with expert-written definitions and examples. * A large data scale, containing 98,591 events and 290,613 arguments obtained with laborious human annotation. * The exhaustive annotation supporting all task variants of EAE, which annotates both entity and non-entity event arguments at the document level.
MultiWOZ-coref, (or MultiWOZ 2.3) is an extension of the MultiWOZ dataset that adds co-reference annotations in addition to corrections of dialogue acts and dialogue states. Source: https://github.com/lexmen318/MultiWOZ_2.3
This project is a collection of three corpora which can be used for evaluating chatbots or other conversational interfaces. Two of the corpora were extracted from StackExchange, one from a Telegram chatbot.
Phrase in Context is a curated benchmark for phrase understanding and semantic search, consisting of three tasks of increasing difficulty: Phrase Similarity (PS), Phrase Retrieval (PR) and Phrase Sense Disambiguation (PSD). The datasets are annotated by 13 linguistic experts on Upwork and verified by two groups: ~1000 AMT crowdworkers and another set of 5 linguistic experts. PiC benchmark is distributed under CC-BY-NC 4.0.
We release various types of word embeddings for multiple Indian languages. Please note that for a majority of our work, we had transliterated the corpora to the Devanagiri script and the script is changed. Word Embedding models using FastText, ElMo, and cross-lingual models based on an orthogonal alignment of monolingual models for all pairs of these languages.
Thunder-NUBench (Negation Understanding Benchmark) is a benchmark specifically designed to evaluate large language models’ (LLMs) sentence-level understanding of negation. Thunder-NUBench introduces rich, manually curated sentence pairs and multiple-choice tasks that contrast standard negation with structurally similar distractors (e.g., local negation, contradiction, paraphrase). The goal is to probe semantic-level understanding of negation.
출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.