Papers with Code 아카이브

Natural Language Inference

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
3
평가표
43
보존된 평가 기록
505

Natural language inference (NLI) is the task of determining whether a "hypothesis" is true (entailment), false (contradiction), or undetermined (neutral) given a "premise".

Example:

Premise Label Hypothesis
A man inspects the uniform of a figure in some East Asian country. contradiction The man is sleeping.
An older and younger man smiling. neutral Two men are smiling and laughing at the cats playing on the floor.
A soccer game with multiple males playing. entailment Some men are playing a sport.

Approaches used for NLI include earlier symbolic and statistical approaches to more recent deep learning approaches. Benchmark datasets used for NLI include SNLI, MultiNLI, SciTail, among others. You can get hands-on practice on the SNLI task by following this d2l.ai chapter.

Further readings:

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

IndicGLUE

We now introduce IndicGLUE, the Indic General Language Understanding Evaluation Benchmark, which is a collection of various NLP tasks as de- scribed below. The goal is to provide an evaluation benchmark for natural language understanding ca- pabilities of NLP models on diverse tasks and mul- tiple Indian languages.

WNLI

The WNLI dataset is a part of the GLUE benchmark used for Natural Language Inference (NLI). It contains pairs of sentences, and the task is to determine whether the second sentence is an entailment of the first one or not. The dataset is used to train and evaluate models on their ability to understand these relationships between sentences.

KorSTS

KorSTS is a dataset for semantic textural similarity (STS) in Korean. The dataset is constructed by automatically the STS-B dataset. To ensure translation quality, two professional translators with at least seven years of experience who specialize in academic papers/books as well as business contracts post-edited a half of the dataset each and cross-checked each other’s translation afterward. The KorSTS dataset comprises 5,749 training examples translated automatically and 2,879 evaluation examples translated manually. Source: https://github.com/kakaobrain/KorNLUDatasets Image Source: https://github.com/kakaobrain/KorNLUDatasets

Story Commonsense

Story Commonsense is a new large-scale dataset with rich low-level annotations and establishes baseline performance on several new tasks, suggesting avenues for future research. Source: Modeling Naive Psychology of Characters in Simple Commonsense Stories

CICERO

CICERO contains 53,000 inferences for five commonsense dimensions -- cause, subsequent event, prerequisite, motivation, and emotional reaction -- collected from 5600 dialogues. It involves two challenging generative and multi-choice alternative selection tasks for the state-of-the-art NLP models to solve. Download the dataset using this link.

FLUE

FLUE is a French Language Understanding Evaluation benchmark. It consists of 5 tasks: Text Classification, Paraphrasing, Natural Language Inference, Constituency Parsing and Part-of-Speech Tagging, and Word Sense Disambiguation.

TaxiNLI

TaxiNLI is a dataset collected based on the principles and categorizations of the aforementioned taxonomy. A subset of examples are curated from MultiNLI (Williams et al., 2018) by sampling uniformly based on the entailment label and the domain. The dataset is annotated with finegrained category labels. Source: TaxiNLI: Taking a Ride up the NLU Hill

CommitmentBank

The CommitmentBank is a corpus of 1,200 naturally occurring discourses whose final sentence contains a clause-embedding predicate under an entailment canceling operator (question, modal, negation, antecedent of conditional).

ART Dataset

ART consists of over 20k commonsense narrative contexts and 200k explanations.

FarsTail

Natural Language Inference (NLI), also called Textual Entailment, is an important task in NLP with the goal of determining the inference relationship between a premise p and a hypothesis h. It is a three-class problem, where each pair (p, h) is assigned to one of these classes: "ENTAILMENT" if the hypothesis can be inferred from the premise, "CONTRADICTION" if the hypothesis contradicts the premise, and "NEUTRAL" if none of the above holds. There are large datasets such as SNLI, MNLI, and SciTail for NLI in English, but there are few datasets for poor-data languages like Persian. Persian (Farsi) language is a pluricentric language spoken by around 110 million people in countries like Iran, Afghanistan, and Tajikistan. FarsTail is the first relatively large-scale Persian dataset for NLI task. A total of 10,367 samples are generated from a collection of 3,539 multiple-choice questions. The train, validation, and test portions include 7,266, 1,537, and 1,564 instances, respectively. Source: https://github.com/dml-qom/FarsTail Image Source: https://github.com/dml-qom/FarsTail

MedNLI

The MedNLI dataset consists of the sentence pairs developed by Physicians from the Past Medical History section of MIMIC-III clinical notes annotated for Definitely True, Maybe True and Definitely False. The dataset contains 11,232 training, 1,395 development and 1,422 test instances. This provides a natural language inference task (NLI) grounded in the medical history of patients. Source: MT-Clinical BERT: Scaling Clinical Information Extraction with Multitask Learning Image Source: https://arxiv.org/abs/1904.02181

SciTail

The SciTail dataset is an entailment dataset created from multiple-choice science exams and web sentences. Each question and the correct answer choice are converted into an assertive statement to form the hypothesis. We use information retrieval to obtain relevant text from a large text corpus of web sentences, and use these sentences as a premise P. We crowdsource the annotation of such premise-hypothesis pair as supports (entails) or not (neutral), in order to create the SciTail dataset. The dataset contains 27,026 examples with 10,101 examples with entails label and 16,925 examples with neutral label. Source: Allen Institute for AI Image source: Allen Institute for AI

SelQA

SelQA is a dataset that consists of questions generated through crowdsourcing and sentence length answers that are drawn from the ten most prevalent topics in the English Wikipedia. Source: SelQA: A New Benchmark for Selection-based Question Answering

DocNLI

DocNLI is a large-scale dataset for document-level NLI. DocNLI is transformed from a broad range of NLP problems and covers multiple genres of text. The premises always stay in the document granularity, whereas the hypotheses vary in length from single sentences to passages with hundreds of words. Additionally, DocNLI has pretty limited artifacts which unfortunately widely exist in some popular sentence-level NLI datasets.

DaNetQA

DaNetQA is a question answering dataset for yes/no questions. These questions are naturally occurring ---they are generated in unprompted and unconstrained settings. Each example is a triplet of (question, passage, answer), with the title of the page as optional additional context. The text-pair classification setup is similar to existing natural language inference tasks. By sampling questions from a distribution of information-seeking queries (rather than prompting annotators for text pairs), we observe significantly more challenging examples compared to existing NLI datasets. ### Example { "text": "В период с 1969 по 1972 год по программе «Аполлон» было выполнено 6 полётов с посадкой на Луне. Всего на Луне высаживались 12 астронавтов США. Список космонавтов Список космонавтов — участников орбитальных космических полётов Список астронавтов США — участников орбитальных космических полётов Список космонавтов СССР и России — участников космических полётов Список женщин-космонавтов Список космонавтов, посещавших МКС Энциклопедия астронавтики.", "question": "Был ли человек на луне?", "label": true, "idx": 5 } ### How did we collect data? All text examples were collected in accordance with the methodology for collecting the original dataset. Answers to the questions were received with the help of assessors, and texts were also received automatically using ODQA systems on Wikipedia. Human assessment was carried out on Yandex.Toloka. *Additionally, to increase number of samples and the distribution of yes/no answers, we added extra data in the same format (data were collected from Yandex.Toloka while generating MuSeRC dataset).

JGLUE

JGLUE, Japanese General Language Understanding Evaluation, is built to measure the general NLU ability in Japanese.

KUAKE-QQR

KUAKE Query-Query Relevance, a dataset used to evaluate the relevance of the content expressed in two queries, is used for the KUAKE-QQR task. Similar to KUAKE-QTR, the task aims to estimate query-query relevance, which is an essential and challenging task in real-world search engines.

KUAKE-QTR

KUAKE Query Title Relevance, a dataset used to estimate the relevance of the title of a query document, is used for the KUAKE-QTR task. Given a query (e.g., “Symptoms of vitamin B deficiency”), the task aims to find the relevant title (e.g., “The main manifestations of vitamin B deficiency”).

TERRa

Textual Entailment Recognition has been proposed recently as a generic task that captures major semantic inference needs across many NLP applications, such as Question Answering, Information Retrieval, Information Extraction, and Text Summarization. This task requires to recognize, given two text fragments, whether the meaning of one text is entailed (can be inferred) from the other text. ### Task Type RTE (Recognizing Textual Entailment) Sentence Pair Classification - Entailment - Not Entailment ### Example { "premise": "Автор поста написал в комментарии, что прорвалась канализация.", "hypothesis": "Автор поста написал про канализацию.", "label": "entailment", "idx": "6062" } ### How did we collect data? All text examples were collected from open news sources and literary magazines, then manually reviewed and supplemented by a human assessment on Yandex.Toloka

IMPPRES

An IMPlicature and PRESupposition diagnostic dataset (IMPPRES), consisting of >25k semiautomatically generated sentence pairs illustrating well-studied pragmatic inference types. Source: Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition

RCB

The Russian Commitment Bank is a corpus of naturally occurring discourses whose final sentence contains a clause-embedding predicate under an entailment cancelling operator (question, modal, negation, antecedent of conditional). ### Task Type RTE (Recognizing Textual Entailment) Sentence Pair Classification - Entailment - Contradiction - Neutral ### Example { "premise": "Сумма ущерба составила одну тысячу рублей. Уточняется, что на место происшествия выехала следственная группа, которая установила личность злоумышленника. Им оказался местный житель, ранее судимый за подобное правонарушение.", "hypothesis": "Ранее местный житель совершал подобное правонарушение.", "verb": "судить", "negation": "no_negation", "label": "entailment", "idx": 269 } ### How did we collect data? All text examples were collected from open news sources and literary magazines, then manually reviewed and supplemented by a human assessment on Yandex.Toloka.

LiDiRus

LiDiRus is a diagnostic dataset that covers a large volume of linguistic phenomena, while allowing you to evaluate information systems on a simple test of textual entailment recognition. See more details diagnostics. ### Task Type RTE (Recognizing Textual Entailment) Sentence Pair Classification - Entailment - Not Entailment ### Example { 'sentence1': "Кошка сидела на коврике.", 'sentence2': "Кошка не сидела на коврике.", 'label': 'not_entailment', 'knowledge': '', 'lexical-semantics': '', 'logic': 'Negation', 'predicate-argument-structure': '' } ### How did we collect data? All text examples manually translated and adapted from English SuperGLUE Diagnostics

RuSentRel

RuSentRel is a corpus of analytical articles translated into Russian texts in the domain of international politics obtained from foreign authoritative sources. The collected articles contain both the author's opinion on the subject matter of the article and a large number of references mentioned between the participants of the described situations. In total, 73 large analytical texts were labeled with about 2000 relations. Source: RuSentRel

IndoNLI

IndoNLI is the first human-elicited NLI dataset for Indonesian consisting of nearly 18K sentence pairs annotated by crowd workers and experts.

Mindgames

We generate epistemic reasoning problems using modal logic to target theory of mind (tom) in natural language processing models.

XWINO

XWINO is a multilingual collection of Winograd Schemas in six languages that can be used for evaluation of cross-lingual commonsense reasoning capabilities. The datasets that comprise XWINO are: * Source: The original Winograd Schema Challenge (Levesque, 2012); * Source: Additional data from the SuperGLUE WSC benchmark (Wang et al., 2019); * Source: The Definite Pronoun Resolution dataset (Rahman and Ng, 2012) (accessed from https://github.com/Yre/wsc_naive); * Source: A collection of French Winograd Schemas (Amsili and Seminck, 2017); * Source: Japanese translation of Winograd Schema Challenge (柴田知秀 et al., 2015); * Source: Russian Winograd Schema Challenge (Shavrina et al., 2020); * Source: A collection of Winograd Schemas in Chinese; * Source: Winograd Schemas in Portuguese (Melo et al., 2019).

esXNLI

esXNLI is a bilingual NLI dataset. It comprises 2,490 examples from 5 different genres that were originally annotated in Spanish, and translated into English by professional translators. It serves as a counterpoint to XNLI, which was originally annotated in English and translated into 14 other languages, including Spanish. The dataset was conceived to be used in conjunction with the XNLI development set to analyse the effect of translation in cross-lingual transfer learning. Source: https://github.com/artetxem/esxnli

JUSTICE

The dataset contains 3304 cases from the Supreme Court of the United States from 1955 to 2021. Each case has the case's identifiers as well as the facts of the case and the decision outcome. Other related datasets rarely included the facts of the case which could prove to be helpful in natural language processing applications. One potential use case of this dataset is determining the outcome of a case using its facts.

Pars-ABSA

Pars-ABSA is a manually annotated Persian dataset, Pars-ABSA, which is verified by 3 native Persian speakers. The dataset consists of 5,114 positive, 3,061 negative and 1,827 neutral data samples from 5,602 unique reviews. Source: Pars-ABSA: an Aspect-based Sentiment Analysis dataset for Persian

Tasksource

Huggingface Datasets is a great library, but it lacks standardization, and datasets require preprocessing work to be used interchangeably. tasksource automates this and facilitates reproducible multi-task learning scaling. Each dataset is standardized to either MultipleChoice, Classification, or TokenClassification dataset with identical fields. We do not support generation tasks as they are addressed by promptsource. All implemented preprocessings are in tasks.py or tasks.md. A preprocessing is a function that accepts a dataset and returns the standardized dataset. Preprocessing code is concise and human-readable.

WeiboPolls

Dataset Description The dataset described in the provided text is focused on social media polls collected from Weibo, a popular Chinese microblogging platform. The dataset aims to empirically study social media polls and analyze user engagement patterns. ### Characteristics of the Dataset - Size: The dataset consists of 20,252 polls collected from 1,860 users on Weibo. - Data Collection: The polls were obtained by sampling Weibo posts containing polls and examining the posting history of the authors. The dataset also includes comments on each post. - Sparsity: The dataset faces the challenge of sparse distribution of polls on Weibo, as less than 0.1% of the randomly gathered posts contained polls. - Content: The dataset includes user-generated polls with questions, answer choices, and corresponding votes. The polls often incorporate trendy hashtags to attract user attention and cover various topics, including social events, public emergencies (such as the COVID-19 outbreak), entertainment topics (celebrities, TV shows), and more. ### Motivations and Summary The motivation behind collecting this dataset is to explore social media polls on Weibo and analyze user engagement patterns. The study aims to understand how users interact with polls, the influence of polls on user engagement, and the types of topics that are more likely to contain polls. The dataset provides insights into user behavior on Weibo by examining factors such as the length of posts, comments, questions, and answers. It also highlights the preference for voting over commenting as a means of expressing opinions. The dataset's analysis suggests that posts with polls tend to attract more comments, likes, and reposts compared to posts without polls. ### Potential Use Cases This dataset can be useful for various research and practical applications, such as: - Social Media Analysis: Researchers can analyze the characteristics and dynamics of social media polls, understanding how they influence user engagement and the types of topics that attract poll creation. - User Engagement Studies: The dataset allows for the exploration of user behavior and preferences when it comes to interacting with polls, providing insights into the factors that drive user engagement on social media platforms. - Trend Analysis: By examining the hashtags associated with polls, the dataset can contribute to understanding social events, public emergencies, and entertainment trends on Weibo. - Marketing and Advertising: The dataset can assist marketers and advertisers in understanding user preferences and interests, enabling them to create targeted campaigns based on the popular topics identified in the dataset. Please note that the actual contents and specific applications of the dataset may vary based on further analysis and research conducted by users.

WiLI-2018

WiLI-2018 is a benchmark dataset for monolingual written natural language identification. WiLI-2018 is a publicly available, free of charge dataset of short text extracts from Wikipedia. It contains 1000 paragraphs of 235 languages, totaling in 23500 paragraphs. WiLI is a classification dataset: Given an unknown paragraph written in one dominant language, it has to be decided which language it is. Source: The WiLI benchmark dataset for written language identification

ArNLI

Natural Language Inference processes pairs of sentences to extract their semantic relations. Pair sentences are annotated with three classes (Contradictions, Entailment, Neutral). 1- Entailment: A and B intersect (A ∩ B ≠ ∅, A ∪ B ≠ Universe) or one may contain another ((A ∁ B) OR (B ∁ A)). 2- Contradiction: A, and B cannot be together, if A is True then B is False (A ∩ B = ∅, and A ∪ B = Universe). 3- Neutral: A, and B have no semantic relations. Each one of them is a different set in the universe. No intersection and their union is a subset of the universe, not all universe (A ∩ B = ∅ and A ∪ B ≠Universe)

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.