Papers with Code 아카이브

Question Answering

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
19
평가표
153
보존된 평가 기록
1,781

Question answering can be segmented into domain-specific tasks like community question answering and knowledge-base question answering. Popular benchmark datasets for evaluation question answering systems include SQuAD, HotPotQA, bAbI, TriviaQA, WikiQA, and many others. Models for question answering are typically evaluated on metrics like EM and F1. Some recent top performing models are T5 and XLNet.

( Image credit: SQuAD )

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

MultiTQ

MULTITQ is a large-scale dataset featuring ample relevant facts and multiple temporal granularities.

ProtoQA

ProtoQA is a question answering dataset for training and evaluating common sense reasoning capabilities of artificial intelligence systems in such prototypical situations. The training set is gathered from an existing set of questions played in a long-running international game show FAMILY- FEUD. The hidden evaluation set is created by gathering answers for each question from 100 crowd-workers. Source: https://github.com/iesl/protoqa-data

TIQ

Existing benchmarks for temporal QA focus on a single information source (either a KB or a text corpus), and include only few questions with implicit constraints. we devise a new method for automatically creating temporal questions with implicit constraints, with systematic controllability of different aspects, including the relative importance of different source types (text, infoboxes, KB), fractions of prominent vs. long-tail entities, question complexity, and more.

VNBench

VNBench is a comprehensive benchmark suite for video generative models, which evaluates video generation quality across specific, hierarchical, and disentangled dimensions, each with tailored prompts and evaluation methods. The suite includes 16 dimensions for evaluating Text-to-Video (T2V) models, such as subject consistency, motion smoothness, and overall consistency. VNBench also supports evaluating Image-to-Video (I2V) models and has recently introduced VBench-Long for evaluating longer videos¹. It's designed to align with human perceptions and provide valuable insights for future developments in video generation. # Diversity of “Needle” Types Edit: Using artificially added subtitles as the "needle". These subtitles are embedded in video frames to simulate the scenario of finding specific textual information in a video. Insert: Using images as the "needle". These images are inserted as static segments between video frames to assess the model's ability to recognize and remember static images in a video. Level Classification: Classified into two levels based on image recognizability. The first level uses common objects (e.g., fruit images), while the second level uses more challenging landmark or object images, increasing the task's difficulty. # Diversity of Video “Haystack” Temporal Distribution: The video "haystack" used by VNBench comes from different data sources, with video durations ranging from 10 seconds to 180 seconds. This covers short, medium, and long video lengths to evaluate the model's adaptability to different video lengths. Content Coverage: The video content includes various scenes, ensuring the evaluation's broadness and the diversity of video sources. # Diversity of Queries Retrieval Task: Requires the model to retrieve specific "needles" from videos, assessing the model's fine-grained understanding and information extraction ability. Ordering Task: Requires the model to identify and order the timestamps of all inserted "needles" in the video, assessing the model's understanding of video temporal dynamics and event sequences. Counting Task: Requires the model to count the occurrences of specific objects in the video, including recognizing and tracking repetitive patterns within and across frames. This assesses the model's understanding of spatial and temporal dimensions. (1) Vchitect/VBench: [CVPR2024 Highlight] VBench - GitHub. https://github.com/Vchitect/VBench. (2) VBench: Comprehensive Benchmark Suite for Video Generative Models. https://arxiv.org/abs/2311.17982. (3) Linux中vdbench的安装与使用_vdbench参数详解-CSDN博客. https://blog.csdn.net/SweeNeil/article/details/95338293. (4) Cinebench 2024 Downloads - Maxon. https://www.maxon.net/zh/downloads/cinebench-2024-downloads. (5) undefined. https://doi.org/10.48550/arXiv.2311.17982.

CLEVR-Dialog

CLEVR-Dialog is a large diagnostic dataset for studying multi-round reasoning in visual dialog. Specifically, that authors construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset. This combination results in a dataset where all aspects of the visual dialog are fully annotated. In total, CLEVR-Dialog contains 5 instances of 10-round dialogs for about 85k CLEVR images, totaling to 4.25M question-answer pairs. The CLEVR-Dialog is used to benchmark performance of standard visual dialog models; in particular, on visual coreference resolution (as a function of the coreference distance). This is the first analysis of its kind for visual dialog models that was not possible without this dataset. CLEVR-Dialog is aims to help inform the development of future models for visual dialog. Source: CLEVR-Dialog

FM-IQA

FM-IQA is a question-answering dataset containing over 150,000 images and 310,000 freestyle Chinese question-answer pairs and their English translations. Source: Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering

LAReQA

A challenging new benchmark for language-agnostic answer retrieval from a multilingual candidate pool. Source: LAReQA: Language-agnostic answer retrieval from a multilingual pool

LC-QuAD

LC-QuAD is a Large Question Answering dataset with 30,000 pairs of questions and its corresponding SPARQL query. The target knowledge base is Wikidata and DBpedia, specifically the 2018 version.

Perception Test

Perception Test is a benchmark designed to evaluate the perception and reasoning skills of multimodal models. It introduces real-world videos designed to show perceptually interesting situations and defines multiple tasks that require understanding of memory, abstract patterns, physics, and semantics – across visual, audio, and text modalities. The benchmark consists of 11.6k videos, 23s average length, filmed by around 100 participants worldwide. The videos are densely annotated with six types of labels: object and point tracks, temporal action and sound segments, multiple-choice video question-answers and grounded video question-answers. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or fine tuning regime.

SberQuAD

A large scale analogue of Stanford SQuAD in the Russian language - is a valuable resource that has not been properly presented to the scientific community. Source: SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysis See DeepPavlov link ### Model results | Model config | EM (dev) | F-1 (dev) | |------------------------------|-------------|-------------| |DeepPavlov RuBERT | 66.30+-0.24 | 84.60+-0.11 | | DeepPavlov multilingual BERT | 64.35+-0.39 | 83.39+-0.08 | | DeepPavlov R-Net | 60.62 | 80.04 |

WikiConv

A corpus that encompasses the complete history of conversations between contributors to Wikipedia, one of the largest online collaborative communities. By recording the intermediate states of conversations---including not only comments and replies, but also their modifications, deletions and restorations---this data offers an unprecedented view of online conversation. Source: WikiConv: A Corpus of the Complete Conversational History of a Large Online Collaborative Community

AQUA

The question-answer (QA) pairs are automatically generated using state-of-the-art question generation methods based on paintings and comments provided in an existing art understanding dataset. The QA pairs are cleansed by crowdsourcing workers with respect to their grammatical correctness, answerability, and answers' correctness. The dataset inherently consists of visual (painting-based) and knowledge (comment-based) questions. Source: A Dataset and Baselines for Visual Question Answering on Art

ComQA

ComQA is a large dataset of real user questions that exhibit different challenging aspects such as compositionality, temporal reasoning, and comparisons. ComQA questions come from the WikiAnswers community QA platform, which typically contains questions that are not satisfactorily answerable by existing search engine technology. Source: ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters Image Source: http://qa.mpi-inf.mpg.de/comqa/

CQASUMM

CQASUMM is a dataset for CQA (Community Question Answering) summarization, constructed from the 4.4 million Yahoo! Answers L6 dataset. The dataset contains ~300k annotated samples. Source: CQASUMM

E-KAR

The ability to recognize analogies is fundamental to human cognition. Existing benchmarks to test word analogy do not reveal the underneath process of analogical reasoning of neural models. Holding the belief that models capable of reasoning should be right for the right reasons, we propose a first-of-its-kind Explainable Knowledge-intensive Analogical Reasoning benchmark (E-KAR). Our benchmark consists of 1,655 (in Chinese) and 1,251 (in English) problems sourced from the Civil Service Exams, which require intensive background knowledge to solve. More importantly, we design a free-text explanation scheme to explain whether an analogy should be drawn, and manually annotate them for each and every question and candidate answer.

How2R

Amazon Mechanical Turk (AMT) is used to collect annotations on HowTo100M videos. 30k 60-second clips are randomly sampled from 9,421 videos and present each clip to the turkers, who are asked to select a video segment containing a single, self-contained scene. After this segment selection step, another group of workers are asked to write descriptions for each displayed segment. Narrations are not provided to the workers to ensure that their written queries are based on visual content only. These final video segments are 10-20 seconds long on average, and the length of queries ranges from 8 to 20 words. From this process, 51,390 queries are collected for 24k 60-second clips from 9,371 videos in HowTo100M, on average 2-3 queries per clip. The video clips and its associated queries are split into 80% train, 10% val and 10% test. Source: HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

KnowIT VQA

KnowIT VQA is a video dataset with 24,282 human-generated question-answer pairs about The Big Bang Theory. The dataset combines visual, textual and temporal coherence reasoning together with knowledge-based questions, which need of the experience obtained from the viewing of the series to be answered. Source: KnowIT VQA: Answering Knowledge-Based Questions about Videos

LEAF-QA

LEAF-QA, a comprehensive dataset of 250,000 densely annotated figures/charts, constructed from real-world open data sources, along with ~2 million question-answer (QA) pairs querying the structure and semantics of these charts. LEAF-QA highlights the problem of multimodal QA, which is notably different from conventional visual QA (VQA), and has recently gained interest in the community. Furthermore, LEAF-QA is significantly more complex than previous attempts at chart QA, viz. FigureQA and DVQA, which present only limited variations in chart data. LEAF-QA being constructed from real-world sources, requires a novel architecture to enable question answering. Source: LEAF-QA: Locate, Encode & Attend for Figure Question Answering

RadQA

RadQA is a radiology question answering dataset with 3074 questions posed against radiology reports and annotated with their corresponding answer spans (resulting in a total of 6148 question-answer evidence pairs) by physicians. The questions are manually created using the clinical referral section of the reports that take into account the actual information needs of ordering physicians and eliminate bias from seeing the answer context (and, further, organically create unanswerable questions). The answer spans are marked within the Findings and Impressions sections of a report. The dataset aims to satisfy the complex clinical requirements by including complete (yet concise) answer phrases (which are not just entities) that can span multiple lines.

SelQA

SelQA is a dataset that consists of questions generated through crowdsourcing and sentence length answers that are drawn from the ten most prevalent topics in the English Wikipedia. Source: SelQA: A New Benchmark for Selection-based Question Answering

SPARTQA

SpartQA is a textual question answering benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior datasets and that is challenging for state-of-the-art language models (LM). SPARTQA is built on NLVR’s images containing more objects with richer spatial structures. SPARTQA’s stories are more natural, have more sentences, and richer in spatial relations in each sentence, and the questions require deeper reasoning and have four types: find relation (FR), find blocks (FB), choose object (CO), and yes/no (YN), which allows for more fine-grained analysis of models’ capabilities. https://aclanthology.org/2021.naacl-main.364/

SPARTQA -

We take advantage of the ground truth of NLVR images, design CFGs to generate stories, and use spatial reasoning rules to ask and answer spatial reasoning questions. This automatically generated data is called SpaRTQA. https://aclanthology.org/2021.naacl-main.364/

SQuADShifts

Provides four new test sets for the Stanford Question Answering Dataset (SQuAD) and evaluate the ability of question-answering systems to generalize to new data. Source: The Effect of Natural Distribution Shift on Question Answering Models

CLEVR-Math

CLEVR-Math is a multi-modal math word problems dataset consisting of simple math word problems involving addition/subtraction, represented partly by a textual description and partly by an image illustrating the scenario. These word problems requires a combination of language, visual and mathematical reasoning.

GermanQuAD

GermanQuAD is a Question Answering (QA) dataset of 13,722 extractive question/answer pairs in German.

OPIEC

OPIEC is an Open Information Extraction (OIE) corpus, constructed from the entire English Wikipedia. It containing more than 341M triples. Each triple from the corpus is composed of rich meta-data: each token from the subj / obj / rel along with NLP annotations (POS tag, NER tag, ...), provenance sentence (along with its dependency parse, sentence order relative to the article), original (golden) links contained in the Wikipedia articles, space / time. Source: OPIEC

PEYMA

Peyma is a Persian NER dataset to train and test NER systems. It is constructed by collecting documents from ten news websites.

QUASAR-S

QUASAR-S is a large-scale dataset aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. It consists of 37,362 cloze-style (fill-in-the-gap) queries constructed from definitions of software entity tags on the popular website Stack Overflow. The posts and comments on the website serve as the background corpus for answering the cloze questions. The answer to each question is restricted to be another software entity, from an output vocabulary of 4874 entities. Source: Quasar: Datasets for Question Answering by Search and Reading Image Source: Quasar: Datasets for Question Answering by Search and Reading

Quizbowl

Consists of multiple sentences whose clues are arranged by difficulty (from obscure to obvious) and uniquely identify a well-known entity such as those found on Wikipedia. Source: Quizbowl: The Case for Incremental Question Answering

SubjQA

SubjQA is a question answering dataset that focuses on subjective (as opposed to factual) questions and answers. The dataset consists of roughly 10,000 questions over reviews from 6 different domains: books, movies, grocery, electronics, TripAdvisor (i.e. hotels), and restaurants. Each question is paired with a review and a span is highlighted as the answer to the question (with some questions having no answer). Moreover, both questions and answer spans are assigned a subjectivity label by annotators. Questions such as "How much does this product weigh?" is a factual question (i.e., low subjectivity), while "Is this easy to use?" is a subjective question (i.e., high subjectivity). Source: https://github.com/megagonlabs/SubjQA

COG

A configurable visual question and answer dataset (COG) to parallel experiments in humans and animals. COG is much simpler than the general problem of video analysis, yet it addresses many of the problems relating to visual and logical reasoning and memory -- problems that remain challenging for modern deep learning architectures. Source: A Dataset and Architecture for Visual Reasoning with a Working Memory

DaNetQA

DaNetQA is a question answering dataset for yes/no questions. These questions are naturally occurring ---they are generated in unprompted and unconstrained settings. Each example is a triplet of (question, passage, answer), with the title of the page as optional additional context. The text-pair classification setup is similar to existing natural language inference tasks. By sampling questions from a distribution of information-seeking queries (rather than prompting annotators for text pairs), we observe significantly more challenging examples compared to existing NLI datasets. ### Example { "text": "В период с 1969 по 1972 год по программе «Аполлон» было выполнено 6 полётов с посадкой на Луне. Всего на Луне высаживались 12 астронавтов США. Список космонавтов Список космонавтов — участников орбитальных космических полётов Список астронавтов США — участников орбитальных космических полётов Список космонавтов СССР и России — участников космических полётов Список женщин-космонавтов Список космонавтов, посещавших МКС Энциклопедия астронавтики.", "question": "Был ли человек на луне?", "label": true, "idx": 5 } ### How did we collect data? All text examples were collected in accordance with the methodology for collecting the original dataset. Answers to the questions were received with the help of assessors, and texts were also received automatically using ODQA systems on Wikipedia. Human assessment was carried out on Yandex.Toloka. *Additionally, to increase number of samples and the distribution of yes/no answers, we added extra data in the same format (data were collected from Yandex.Toloka while generating MuSeRC dataset).

DBLP-QuAD

In this work we create a question answering dataset over the DBLP scholarly knowledge graph (KG). DBLP is an on-line reference for bibliographic information on major computer science publications that indexes over 4.4 million publications, published by more than 2.2 million authors. Our dataset consists of 10,000 question answer pairs with the corresponding SPARQL queries which can be executed over the DBLP KG to fetch the correct answer. To the best of our knowledge, this is the first QA dataset for scholarly KGs.

ForecastQA

ForecastQA is a question-answering dataset consisting of 10,392 event forecasting questions, which have been collected and verified via crowdsourcing efforts. The forecasting problem for this dataset is formulated as a restricted-domain, multiple-choice, question-answering (QA) task that simulates the forecasting scenario. Source: ForecastQA: A Question Answering Challenge for Event Forecasting

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.