Papers with Code · CC BY-SA 4.0

데이터셋 탐색

데이터셋의 설명·홈페이지를 찾고, 관련 태스크의 평가 기록을 살펴보세요.

관련 논문 48편

Textbook Question Answering

TQA
TQA 설명 보기

The TextbookQuestionAnswering (TQA) dataset is drawn from middle school science curricula. It consists of 1,076 lessons from Life Science, Earth Science and Physical Science textbooks. This includes 26,260 questions, including 12,567 that have an accompanying diagram. The TQA dataset encourages work on the task of Multi-Modal Machine Comprehension (M3C) task. The M3C task builds on the popular Visual Question Answering (VQA) and Machine Comprehension (MC) paradigms by framing question answering as a machine comprehension task, where the context needed to answer questions is provided and composed of both text and images. The dataset constructed to showcase this task has been built from a middle school science curriculum that pairs a given question to a limited span of knowledge needed to answer it. Source: Allen Institute for AI

관련 논문 933편

HotpotQA

HotpotQA 설명 보기

HotpotQA is a question answering dataset collected on the English Wikipedia, containing about 113K crowd-sourced questions that are constructed to require the introduction paragraphs of two Wikipedia articles to answer. Each question in the dataset comes with the two gold paragraphs, as well as a list of sentences in these paragraphs that crowdworkers identify as supporting facts necessary to answer the question. A diverse range of reasoning strategies are featured in HotpotQA, including questions involving missing entities in the question, intersection questions (What satisfies property A and property B?), and comparison questions, where two entities are compared by a common attribute, among others. In the few-document distractor setting, the QA models are given ten paragraphs in which the gold paragraphs are guaranteed to be found; in the open-domain fullwiki setting, the models are only given the question and the entire Wikipedia. Models are evaluated on their answer accuracy and explainability, where the former is measured as overlap between the predicted and gold answers with exact match (EM) and unigram F1, and the latter concerns how well the predicted supporting fact sentences match human annotation (Supporting Fact EM/F1). A joint metric is also reported on this dataset, which encourages systems to perform well on both tasks simultaneously. Source: Answering Complex Open-domain Questions Through Iterative Query Generation Image Source: Yang et al

관련 논문 278편

ChartQA

ChartQA 설명 보기

Charts are very popular for analyzing data. When exploring charts, people often ask a variety of complex reasoning questions that involve several logical and arithmetic operations. They also commonly refer to visual features of a chart in their questions. However, most existing datasets do not focus on such complex reasoning questions as their questions are template-based and answers come from a fixed-vocabulary. In this work, we present a large-scale benchmark covering 9.6K human-written questions as well as 23.1K questions generated from human-written chart summaries. To address the unique challenges in our benchmark involving visual and logical reasoning over charts, we present two transformer-based models that combine visual features and the data table of the chart in a unified way to answer questions. While our models achieve the state-of-the-art results on the previous datasets as well as on our benchmark, the evaluation also reveals several challenges in answering complex reasoning questions.

관련 논문 50편

PlotQA

PlotQA 설명 보기

PlotQA is a VQA dataset with 28.9 million question-answer pairs grounded over 224,377 plots on data from real-world sources and questions based on crowd-sourced question templates. Existing synthetic datasets (FigureQA, DVQA) for reasoning over plots do not contain variability in data labels, real-valued data, or complex reasoning questions. Consequently, proposed models for these datasets do not fully address the challenge of reasoning over plots. In particular, they assume that the answer comes either from a small fixed size vocabulary or from a bounding box within the image. However, in practice this is an unrealistic assumption because many questions require reasoning and thus have real valued answers which appear neither in a small fixed size vocabulary nor in the image. In this work, we aim to bridge this gap between existing datasets and real world plots by introducing PlotQA. Further, 80.76% of the out-of-vocabulary (OOV) questions in PlotQA have answers that are not in a fixed vocabulary.

관련 논문 27편

IntentQA

IntentQA 설명 보기

We contribute an IntentQA dataset with diverse intents in daily social activities. We utilize NExT-QA as the source dataset to construct our dataset. NExT-QA dataset is a comprehensive VideoQA dataset with rich natural daily social activities and detailed QA annotations. Originally, the NExT-QA dataset categorizes itself into three types, i.e., Causal, Temporal, Descriptive. We select the inference QA types, i.e., Causal and Temporal, rather than the factoid Descriptive, to build our IntentQA dataset. Particularly, we select both the Causal Why and Causal How subtypes under Causal, and the Temporal Previous and Temporal Next subtypes under Temporal. The Causal Why (CW) QA usually takes the form of ‘Why [action]? For [intent]’, with the key action appearing in the question and the intent in the answer. On the contrary, the Causal How (CH) QA usually takes the form of ‘How [intent]? By [action]’, with the key action appearing in the answer and the intent in the question. The Temporal Previous (TP) QA usually takes the form of ‘What [action A] before [action B]? ’, while the Temporal Next (TN) QA takes the form of ‘What [action B] after [action A]? ’. In the TP&TN QA, the intent is not explicitly expressed in the question nor answer, but is the implicit causal factor linking the two sequential actions.

관련 논문 19편

TweetQA

TweetQA 설명 보기

With social media becoming increasingly popular on which lots of news and real-time events are reported, developing automated question answering systems is critical to the effectiveness of many applications that rely on real-time knowledge. While previous question answering (QA) datasets have concentrated on formal text like news and Wikipedia, the first large-scale dataset for QA over social media data is presented. To make sure the tweets are meaningful and contain interesting information, tweets used by journalists to write news articles are gathered. Then human annotators are asked to write questions and answers upon these tweets. Unlike other QA datasets like SQuAD in which the answers are extractive, the answer are allowed to be abstractive. The task requires model to read a short tweet and a question and outputs a text phrase (does not need to be in the tweet) as the answer. Source: TWEETQA: A Social Media Focused Question Answering Dataset

관련 논문 10편

PointQA

PointQA 설명 보기

PointQA is a set of datasets for Visual Question Datasets (VQA) that require a pointer to an object in the image to be answered correctly. The different datasets are: PointQA-Local, PointQA-LookTwice and PointQA-General.

관련 논문 9편

SPAtial Reasoning on Textual Question Answering

SPARTQA
SPARTQA 설명 보기

SpartQA is a textual question answering benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior datasets and that is challenging for state-of-the-art language models (LM). SPARTQA is built on NLVR’s images containing more objects with richer spatial structures. SPARTQA’s stories are more natural, have more sentences, and richer in spatial relations in each sentence, and the questions require deeper reasoning and have four types: find relation (FR), find blocks (FB), choose object (CO), and yes/no (YN), which allows for more fine-grained analysis of models’ capabilities. https://aclanthology.org/2021.naacl-main.364/

관련 논문 7편

Yes/no Question Answering Dataset for the Russian

DaNetQA
DaNetQA 설명 보기

DaNetQA is a question answering dataset for yes/no questions. These questions are naturally occurring ---they are generated in unprompted and unconstrained settings. Each example is a triplet of (question, passage, answer), with the title of the page as optional additional context. The text-pair classification setup is similar to existing natural language inference tasks. By sampling questions from a distribution of information-seeking queries (rather than prompting annotators for text pairs), we observe significantly more challenging examples compared to existing NLI datasets. ### Example { "text": "В период с 1969 по 1972 год по программе «Аполлон» было выполнено 6 полётов с посадкой на Луне. Всего на Луне высаживались 12 астронавтов США. Список космонавтов Список космонавтов — участников орбитальных космических полётов Список астронавтов США — участников орбитальных космических полётов Список космонавтов СССР и России — участников космических полётов Список женщин-космонавтов Список космонавтов, посещавших МКС Энциклопедия астронавтики.", "question": "Был ли человек на луне?", "label": true, "idx": 5 } ### How did we collect data? All text examples were collected in accordance with the methodology for collecting the original dataset. Answers to the questions were received with the help of assessors, and texts were also received automatically using ODQA systems on Wikipedia. Human assessment was carried out on Yandex.Toloka. *Additionally, to increase number of samples and the distribution of yes/no answers, we added extra data in the same format (data were collected from Yandex.Toloka while generating MuSeRC dataset).

관련 논문 7편

ForecastQA

ForecastQA 설명 보기

ForecastQA is a question-answering dataset consisting of 10,392 event forecasting questions, which have been collected and verified via crowdsourcing efforts. The forecasting problem for this dataset is formulated as a restricted-domain, multiple-choice, question-answering (QA) task that simulates the forecasting scenario. Source: ForecastQA: A Question Answering Challenge for Event Forecasting

관련 논문 5편

ImplicitQA

ImplicitQA 설명 보기

The ImplicitQA dataset was introduced in the paper ImplicitQA: Going beyond frames towards Implicit Video Reasoning. Project page: https://swetha5.github.io/ImplicitQA/ ImplicitQA is a novel benchmark specifically designed to test models on implicit reasoning in Video Question Answering (VideoQA). Unlike existing VideoQA benchmarks that primarily focus on questions answerable through explicit visual content (actions, objects, events directly observable within individual frames or short clips), ImplicitQA addresses the need for models to infer motives, causality, and relationships <u>across discontinuous frames</u>. This mirrors human-like understanding of creative and cinematic videos, which often employ storytelling techniques that deliberately omit certain depictions. The dataset comprises 1,000 meticulously annotated QA pairs derived from over 320 high-quality creative video clips. These QA pairs are systematically categorized into key reasoning dimensions, including: * Lateral spatial reasoning * Vertical spatial reasoning * Relative Depth and proximity * Viewpoint and visibility * Motion and trajectory Dynamics * Causal and motivational reasoning * Social interactions and Relationships * Physical and Environmental context * Inferred counting The annotations are deliberately challenging, crafted to ensure high quality and to highlight the difficulty of implicit reasoning for current VideoQA models.

관련 논문 4편

UniProtQA

UniProtQA 설명 보기

UniProtQA consists of proteins and textual queries about their functions and properties. The dataset is constructed from UniProt, and consists 4 types of questions with regard to functions, official names, protein families, and sub-cellular locations. We collect a total of 569, 516 proteins and 1, 891, 506 question-answering samples.

관련 논문 3편

ConcurrentQA Benchmark

ConcurrentQA Benchmark 설명 보기

ConcurrentQA is a textual multi-hop QA benchmark to require concurrent retrieval over multiple data-distributions (i.e. Wikipedia and email data). The dataset follow the exact same schema and design as HotpotQA. The data set is downloadable here: https://github.com/facebookresearch/concurrentqa. It also contains model and result analysis code. This benchmark can also be used to study privacy when reasoning over data distributed in multiple privacy scopes --- i.e. Wikipedia in the public domain and emails in the private domain. The following is a blog post about the benchmark: https://ai.facebook.com/blog/building-systems-to-reason-securely-over-private-data/

관련 논문 3편

FanOutQA

FanOutQA 설명 보기

FanOutQA is a high quality, multi-hop, multi-document benchmark for large language models using English Wikipedia as its knowledge base. Compared to other question-answering benchmarks, FanOutQA requires reasoning over a greater number of documents, with the benchmark's main focus being on the titular fan-out style of question. We present these questions in three tasks -- closed-book, open-book, and evidence-provided -- which measure different abilities of LLM systems.

관련 논문 1편

ComTQA

ComTQA 설명 보기

The ComTQA dataset is a visual table question answering benchmark. It includes images collected from FinTabNet and PubTables-1M, comprising a total of 9,070 QA pairs with 1,591 images. The dataset is designed to address tasks related to table question answering and is available in English. It falls under the size category of 1K<n<10K and is licensed under cc-by-nc-4.0¹. This dataset is particularly useful for developing and testing algorithms that can interpret and answer questions based on tabular data. It's a valuable resource for researchers and practitioners in the field of machine learning and natural language processing, especially those focusing on the intersection of visual data interpretation and question answering systems¹. (1) ByteDance/ComTQA · Datasets at Hugging Face. https://huggingface.co/datasets/ByteDance/ComTQA. (2) ComQA Dataset | Papers With Code. https://paperswithcode.com/dataset/comqa. (3) COMETA: the corpus of online medical entities - GitHub. https://github.com/cambridgeltl/cometa. (4) COMETA Dataset | Papers With Code. https://paperswithcode.com/dataset/cometa.

관련 논문 1편

MedDistractQA-Bystander

MedDistractQA-Bystander 설명 보기

Novel benchmark adapted from the MedQA, with confounding statements introduced within the question regarding an irrelevant third party.

관련 논문 1편

MedDistractQA-Nonliteral

MedDistractQA-Nonliteral 설명 보기

Novel benchmark adapted from the MedQA, with confounding statements introduced within the question regarding an irrelevant clinical term used in a nonclinical context (e.g., The patient's Zodiac sign is Cancer).

관련 논문 1편

NextQA

NextQA 설명 보기

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

관련 논문 1편

Visual Text Question Answering

VTQA
VTQA 설명 보기

VTQA is a dataset containing open-ended questions about image-text pairs. This dataset requires the model to align multimedia representations of the same entity to implement multi-hop reasoning between image and text and finally use natural language to answer the question. The aim of this dataset is to develop and benchmark models that are capable of multimedia entity alignment, multi-step reasoning and open-ended answer generation. VTQA dataset consists of 10,238 image-text pairs and 27,317 questions. The images are real images from MSCOCO dataset, containing a variety of entities. The annotators are required to first annotate relevant text according to the image, and then ask questions based on the image-text pair, and finally answer the question open-ended.