SequentialQA
SQA 설명 보기
The SQA dataset was created to explore the task of answering sequences of inter-related questions on HTML tables. It has 6,066 sequences with 17,553 questions in total. Source: SQA
데이터셋의 설명·홈페이지를 찾고, 관련 태스크의 평가 기록을 살펴보세요.
The SQA dataset was created to explore the task of answering sequences of inter-related questions on HTML tables. It has 6,066 sequences with 17,553 questions in total. Source: SQA
The CommonsenseQA is a dataset for commonsense question answering task. The dataset consists of 12,247 questions with 5 choices each. The dataset was generated by Amazon Mechanical Turk workers in the following process (an example is provided in parentheses): 1. a crowd worker observes a source concept from ConceptNet (“River”) and three target concepts (“Waterfall”, “Bridge”, “Valley”) that are all related by the same ConceptNet relation (“AtLocation”), 2. the worker authors three questions, one per target concept, such that only that particular target concept is the answer, while the other two distractor concepts are not, (“Where on a river can you hold a cup upright to catch water on a sunny day?”, “Where can I stand on a river to see water falling without getting wet?”, “I’m crossing the river, my feet are wet but my body is dry, where am I?”) 3. for each question, another worker chooses one additional distractor from Concept Net (“pebble”, “stream”, “bank”), and the author another distractor (“mountain”, “bottom”, “island”) manually. Source: CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge Image Source: CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
The NewsQA dataset is a crowd-sourced machine reading comprehension dataset of 120,000 question-answer pairs. * Documents are CNN news articles. * Questions are written by human users in natural language. * Answers may be multiword passages of the source text. * Questions may be unanswerable. * NewsQA is collected using a 3-stage, siloed process. * Questioners see only an article’s headline and highlights. * Answerers see the question and the full article, then select an answer passage. * Validators see the article, the question, and a set of answers that they rank. * NewsQA is more natural and more challenging than previous datasets. Source: https://www.microsoft.com/en-us/research/project/newsqa-dataset/ Image Source: Trischler et al
CosmosQA is a large-scale dataset of 35.6K problems that require commonsense-based reading comprehension, formulated as multiple-choice questions. It focuses on reading between the lines over a diverse collection of people’s everyday narratives, asking questions concerning on the likely causes or effects of events that require reasoning beyond the exact text spans in the context. Source: Teaching Pretrained Models with Commonsense Reasoning: A Preliminary KB-Based Approach Image Source: https://arxiv.org/pdf/1909.00277.pdf
SQA3D is a dataset for embodied scene understanding, where an agent needs to comprehend the scene it situates from an first person's perspective and answer questions. The questions are designed to be situated, embodied and knowledge-intensive. We offer three different modalities to represent a 3D scene: 3D scan, egocentric video and BEV picture.
Contains around 200K dialogs with a total of 1.6M turns. Further, unlike existing large scale QA datasets which contain simple questions that can be answered from a single tuple, the questions in the dialogs require a larger subgraph of the KG. Source: Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph
CoSQA (Code Search and Question Answering) It includes 20,604 labels for pairs of natural language queries and codes, each annotated by at least 3 human annotators.
X-CSQA is a multilingual dataset for Commonsense reasoning research, based on CSQA.
MedConceptsQA - Open Source Medical Concepts QA Benchmark The benchmark can be found here: https://huggingface.co/datasets/ofir408/MedConceptsQA
UIT-ViNewsQA is a new corpus for the Vietnamese language to evaluate healthcare reading comprehension models. The corpus comprises 22,057 human-generated question-answer pairs. Crowd-workers create the questions and their answers based on a collection of over 4,416 online Vietnamese healthcare news articles, where the answers comprise spans extracted from the corresponding articles. Source: New Vietnamese Corpus for Machine ReadingComprehension of Health News Articles
DangerousQA refers to a set of harmful questions used to evaluate the safety and behavior of large language models (LLMs) in generating responses. In the context of the RED-EVAL safety benchmark, DangerousQA consists of 200 harmful questions collected from various sources, such as those related to racism, stereotypes, sexism, legality, toxicity, and harm. These questions are used to test the ability of LLMs to handle sensitive and potentially harmful content and to assess their performance in generating appropriate responses to such prompts.
The ODSQA dataset is a spoken dataset for question answering in Chinese. It contains more than three thousand questions from 20 different speakers. Source: https://github.com/chiahsuan156/ODSQA
Multi-modal situated reasoning in 3D scenes
ACCORD CSQA is an extension of the popular CommonsenseQA (CSQA) dataset using ACCORD, a scalable framework for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs) through controlled, multi-hop counterfactuals. ACCORD closes the measurability gap between commonsense and formal reasoning tasks for LLMs. A detailed understanding of LLMs' commonsense reasoning abilities is severely lagging compared to our understanding of their formal reasoning abilities, since commonsense benchmarks are difficult to construct in a manner that is rigorously quantifiable. Specifically, prior commonsense reasoning benchmarks and datasets are limited to one- or two-hop reasoning or include an unknown (i.e., non-measurable) number of reasoning hops and/or distractors. Arbitrary scalability via compositional construction is also typical of formal reasoning tasks but lacking in commonsense reasoning. Finally, most prior commonsense benchmarks either are limited to a single reasoning skill or do not control skills. ACCORD aims to address all these gaps by introducing formal elements to commonsense reasoning to explicitly control and quantify reasoning complexity beyond the typical 1 or 2 reasoning hops. Uniquely, ACCORD can automatically generate benchmarks of arbitrary reasoning complexity, and so it scales with future LLM improvements. ACCORD CSQA is a benchmark suite comprising problem with 6 levels of reasoning difficulty, ACCORD CSQA 0 to ACCORD CSQA 5. Experiments on state-of-the-art LLMs show performance degrading to random chance with only moderate scaling, leaving substantial headroom for improvement.
Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset
The goal of InfoLossQA is to generate a series of QA pairs that reveal to lay readers what information a simplified text lacks compared to its original. We provide an annotated dataset in the domain of medical text simplification, specifically abstracts of Randomized Controlled Trials (RCTs). The abstracts were automatically simplified by an LLM (GPT-4). Then, three linguists annotated information loss and wrote the QA pairs. For more details about the dataset, see the project website: https://InfoLossQA.ikim.nrw/#/
The NISQA Corpus includes more than 14,000 speech samples with simulated (e.g. codecs, packet-loss, background noise) and live (e.g. mobile phone, Zoom, Skype, WhatsApp) conditions. Each file is labelled with subjective ratings of the overall quality and the quality dimensions Noisiness, Coloration, Discontinuity, and Loudness. In total, it contains more than 97,000 human ratings for each of the dimensions and the overall MOS. The NISQA Speech Quality Corpus contains two training, two validation and four test datasets: - NISQA_TRAIN_SIM and NISQA_VAL_SIM: contains simulated distortions with speech samples from four different datasets. Divided into a training and a validation set. - NISQA_TRAIN_LIVE and NISQA_VAL_LIVE: contains live phone and Skype recordings with Librivox audiobook samples. Divided into training and validation set. - NISQA_TEST_LIVETALK: contains recordings of real phone and VoIP calls. - NISQA_TEST_FOR: contains live and simulated conditions with speech samples from the forensic speech dataset. - NISQA_TEST_NSC: contains live and simulated conditions with speech samples from the NSC dataset. - NISQA_TEST_P501: contains live and simulated conditions with speech samples from ITU-T Rec. P.501. The datasets are provided under the original terms of the used source speech and noise samples. Please see the individual readme and license files in each of the dataset folders within the NISQA_Corpus.zip for more details about the datasets and the licenses. Generally, all of the files in this corpus can be used for non-commercial research purposes and some of the datasets can be also be used for commercial purposes.