Papers with Code 아카이브

Question Answering

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
19
평가표
153
보존된 평가 기록
1,781

Question answering can be segmented into domain-specific tasks like community question answering and knowledge-base question answering. Popular benchmark datasets for evaluation question answering systems include SQuAD, HotPotQA, bAbI, TriviaQA, WikiQA, and many others. Models for question answering are typically evaluated on metrics like EM and F1. Some recent top performing models are T5 and XLNet.

( Image credit: SQuAD )

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

Researchy Questions

Researchy Questions is a set of about 100k Bing queries that users spent the most effort on. After a labor-intensive filtering funnel from billions of queries, these "needles in the haystack" are non-factoid, multi-perspective questions that require a lot of sub-questions and research to answer adequately. These questions are shown to be harder than other open domain QA datasets like Natural Questions.

ResQ

ReSQ is a real-world Spatial Question Answering dataset with human-generated questions built on an existing corpus with SpRL annotations. This dataset can be used to evaluate spatial language processing models in realistic situations.

Shmoop Corpus

Shmoop Corpus is a dataset of 231 stories that are paired with detailed multi-paragraph summaries for each individual chapter (7,234 chapters), where the summary is chronologically aligned with respect to the story chapter. From the corpus, a set of common NLP tasks are constructed, including Cloze-form question answering and a simplified form of abstractive summarization, as benchmarks for reading comprehension on stories. Source: Shmoop Corpus

Shot2Story20K

A short clip of video may contain progression of multiple events and an interesting story line. A human needs to capture both the event in every shot and associate them together to understand the story behind it. In this work, we present a new multi-shot video understanding benchmark Shot2Story with detailed shot-level captions and comprehensive video summaries. To facilitate better semantic understanding of videos, we provide captions for both visual signals and human narrations. We design several distinct tasks including single-shot video and narration captioning, multi-shot video summarization, and video retrieval with shot descriptions. Preliminary experiments show some challenges to generate a long and comprehensive video summary. Nevertheless, the generated imperfect summaries can already significantly boost the performance of existing video understanding tasks such as video question-answering, promoting an underexplored setting of video understanding with detailed summaries.

TextBox 2.0

TextBox 2.0 is a comprehensive and unified library for text generation, focusing on the use of pre-trained language models (PLMs). The library covers 13 common text generation tasks and their corresponding 83 datasets and further incorporates 45 PLMs covering general, translation, Chinese, dialogue, controllable, distilled, prompting, and lightweight PLMs. Source: TextBox 2.0: A Text Generation Library with Pre-trained Language Models Image Source: https://arxiv.org/pdf/2212.13005v1.pdf

TriviaHG

Nowadays, individuals tend to engage in dialogues with Large Language Models, seeking answers to their questions. In times when such answers are readily accessible to anyone, the stimulation and preservation of human’s cognitive abilities, as well as the assurance of maintaining good reasoning skills by humans becomes crucial. This study addresses such needs by proposing hints (instead of final answers or before giving answers) as a viable solution. We introduce a framework for the automatic hint generation for factoid questions, employing it to construct TriviaHG, a novel large-scale dataset featuring 160,230 hints corresponding to 16,645 questions from the TriviaQA dataset. Additionally, we present an automatic evaluation method that measures the Convergence and Familiarity quality attributes of hints. To evaluate the TriviaHG dataset and the proposed evaluation method, we enlisted 10 individuals to annotate 2,791 hints and tasked 6 humans with answering questions using the provided hints. The effectiveness of hints varied, with success rates of 96%, 78%, and 36% for questions with easy, medium, and hard answers, respectively. Moreover, the proposed automatic evaluation methods showed a robust correlation with annotators’ results. Conclusively, the findings highlight three key insights: the facilitative role of hints in resolving unknown questions, the dependence of hint quality on answer difficulty, and the feasibility of employing automatic evaluation methods for hint assessment.

UIT-ViCoV19QA

The dataset comprises 4,500 question-answer pairs collected from trusted medical sources, with at least one answer and at most four unique paraphrased answers per question

AWS Documentation

We present the AWS documentation corpus, an open-book QA dataset, which contains 25,175 documents along with 100 matched questions and answers. These questions are inspired by the author's interactions with real AWS customers and the questions they asked about AWS services. The data was anonymized and aggregated. All questions in the dataset have a valid, factual and unambiguous answer within the accompanying documents, we deliberately avoided questions that are ambiguous, incomprehensible, opinion-seeking, or not clearly a request for factual information. All questions, answers and accompanying documents in the dataset are annotated by authors. There are two types of answers: text and yes-no-none(YNN) answers. Text answers range from a few words to a full paragraph sourced from a continuous block of words in a document or from different locations within the same document. Every question in the dataset has a matched text answer. Yes-no-none(YNN) answers can be yes, no, or none depending on the type of question. For example the question: “Can I stop a DB instance that has a read replica?” has a clear yes or no answer but the question “What is the maximum number of rows in a dataset in Amazon Forecast?” is not a yes or no question and therefore has a “None” as the YNN answer. 23 questions have ‘Yes’ YNN answers, 10 questions have ‘No’ YNN answers and 67 questions have ‘None’ YNN answers.

CheGeKa

CheGeKa is a Jeopardy!-like Russian QA dataset collected from the official Russian quiz database ChGK. Motivation The task can be considered the most challenging in terms of reasoning, knowledge and logic, as the task implies the QA pairs with a free response form (no answer choices); however, a long chain of causal relationships between facts and associations forms the correct answer. The original corpus of the CheGeKa game was introduced in Mikhalkova (2021). An example in English for illustration purposes: { 'question_id': 3665, 'question': 'THIS MAN replaced John Lennon when the Beatles got together for the last time.', 'answer': 'Julian Lennon', 'topic': 'The Liverpool Four', 'author': 'Bayram Kuliyev', 'tour_name': 'Jeopardy!. Ashgabat-1996', 'tour_link': 'https://db.chgk.info/tour/ash96sv', 'episode': [16], 'perturbation': 'chegeka' } Data Fields - question_id: an integer corresponding to the question id in the database - question: a string containing the question text - answer: a string containing the correct answer to the question - topic: a string containing the question category - author: a string with the full name of the author - tour_name: a string with the title of a tournament - tour link: a string containing the link to a tournament (None for the test set) - perturbation: a string containing the name of the perturbation applied to text. If no perturbation was applied, the dataset name is used - episode: a list of episodes in which the instance is used. Only used for the train set Data Splits The dataset consists of a training set with labeled examples and a test set in two configurations: - raw data: includes the original data with no additional sampling - episodes: data is split into evaluation episodes and includes several perturbations of test for robustness evaluation Test Perturbations Each training episode in the dataset corresponds to seven test variations, including the original test data and six adversarial test sets, acquired through the modification of the original test through the following text perturbations: - ButterFingers: randomly adds noise to data by mimicking spelling mistakes made by humans through character swaps based on their keyboard distance - Emojify: replaces the input words with the corresponding emojis, preserving their original meaning - EDAdelete: randomly deletes tokens in the text - EDAswap: randomly swaps tokens in the text - BackTranslation: generates variations of the context through back-translation (ru -> en -> ru) - AddSent: generates extra words or a sentence at the end of the question

CII-Bench

We introduce the Chinese Image Implication Understanding Benchmark CII-Bench, a new benchmark measuring the higher-order perceptual, reasoning and comprehension abilities of MLLMs when presented with complex Chinese implication images. These images, including abstract artworks, comics and posters, possess visual implications that require an understanding of visual details and reasoning ability. CII-Bench reveals whether current MLLMs, leveraging their inherent comprehension abilities, can accurately decode the metaphors embedded within the complex and abstract information presented in these images. CII-Bench comprises 698 Chinese images, each accompanied by 1 to 3 multiple-choice questions, totaling 800 questions. CII-Bench encompasses images from six distinct domains: Life, Art, Society, Environment, Politics, and Chinese Traditional Culture. It also features a diverse array of image types, including Illustrations, Memes, Posters, Multi-panel Comics, Single-panel Comics, and Paintings.

DART-Math-Hard

🎯 DART-Math > Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving 📝 Paper@arXiv | 🤗 Datasets&Models@HF | 🐱 Code@GitHub 🐦 Thread@X(Twitter) | 🐶 中文博客@知乎 | 📊 Leaderboard@PapersWithCode | 📑 BibTeX ## Datasets: DART-Math DART-Math datasets are the state-of-the-art and data-efficient open-source instruction tuning datasets for mathematical reasoning. DART-Math-Hard contains ~585k mathematical QA pair samples constructed by applying DARS-Prop2Diff to the query set from MATH and GSK8K training sets, achieves SOTA on many challenging mathematical reasoning benchmarks. It introduces a deliberate bias towards hard queries, opposite to vanilla rejection sampling. Performance produced by DART-Math-Hard is usually but not necessarily slightly better (~1% absolutely) than DART-Math-Uniform, which contains ~591k samples constructed by applying DARS-Uniform. ### Comparison between Mathematical Instruction Tuning Datasets Most of previous datasets are constructed with ChatGPT, and many of them are not open-source, especially for ones of the best performance. | Math SFT Dataset | # of Samples | MATH | GSM8K | College | Synthesis Agent(s) | Open-Source | | :--------------------------------------------------------------------------------- | -----------: | -----------------------------------------------------------------: | ---------------------------------------------: | -----------------------------------------------------------------------------------------------------------: | :---------------------- | :-------------------------------------------------------------------------: | | WizardMath | 96k | 32.3 | 80.4 | 23.1 | GPT-4 | ✗ | | MetaMathQA | 395k | 29.8 | 76.5 | 19.3 | GPT-3.5 | ✓ | | MMIQC | 2294k | 37.4 | 75.4 | 28.5 | GPT-4+GPT-3.5+Human | ✓ | | Orca-Math | 200k | -- | -- | -- | GPT-4 | ✓ | | Xwin-Math-V1.1 | 1440k | 45.5 | 84.9 | 27.6 | GPT-4 | ✗ | | KPMath-Plus | 1576k | 46.8 | 82.1 | -– | GPT-4 | ✗ | | MathScaleQA | 2021k | 35.2 | 74.8 | 21.8 | GPT-3.5+Human | ✗ | | DART-Math-Uniform | 591k | 43.5 | 82.6 | 26.9 | DeepSeekMath-7B-RL | ✓ | | DART-Math-Hard | 585k | 45.5 | 81.1 | 29.4 | DeepSeekMath-7B-RL | ✓ | <sup>MATH and GSM8K are in-domain, while College(Math) is out-of-domain. Performance here are of models fine-tuned from Mistral-7B, except for Xwin-Math-V1.1 based on Llama2-7B. Bold/Italic means the best/second best score here.</sup> ## Dataset Construction: DARS - Difficulty-Aware Rejection Sampling Previous works usually synthesize data from proprietary models to augment existing datasets, followed by instruction tuning to achieve top-tier results. However, our analysis of these datasets reveals severe biases towards easy queries, with frequent failures to generate any correct response for the most challenging queries. Motivated by the observation above, we propose to Difficulty-Aware Rejection Sampling (DARS), to collect more responses for more difficult queries. Specifically, we introduce two strategies to increase the number of correct responses for difficult queries: 1) Uniform, which involves sampling responses for each query until each query accumulates $k_u$ correct responses, where $k_u$ is a preset hyperparameter determined by the desired size of the synthetic dataset; 2) Prop2Diff, where we continue sampling responses until the number of correct responses for each query is proportional to its difficulty score. The most challenging queries will receive $k_p$ responses and kp is a hyperparameter. This method introduces a deliberate bias in the opposite direction to vanilla rejection sampling, towards more difficult queries, inspired by previous works that demonstrate difficult samples can be more effective to enhance model capabilities (Sorscher et al., 2022; Liu et al., 2024b). See Figure 1 (Right) for examples of DART-Math-Uniform by DARS-Uniform and DART-Math-Hard by DARS-Prop2Diff. ## Citation If you find our data, model or code useful for your work, please kindly cite our paper: latex @article{tong2024dartmath, title={DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving}, author={Yuxuan Tong and Xiwen Zhang and Rui Wang and Ruidong Wu and Junxian He}, year={2024}, eprint={2407.13690}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2407.13690}, }

Eedi Dataset

The Eedi dataset contains from two school years (September 2018 to May 2020) of students’ answers to mathematics questions from Eedi, a leading educational platform which millions of students interact with daily around the globe. Eedi offers diagnostic questions to students from primary to high school (roughly between 7 and 18 years old). Each diagnostic question is a multiple-choice question with 4 possible answer choices, exactly one of which is correct. Currently, the platform mainly focuses on mathematics questions. The data is split for different tasks: 1 & 2: Answer prediction, 3: Predict question quality, and 4: Recommend questions. The total number of answer records for these tasks training sets exceeds 17 million, making it one of the largest educational datasets to date. We also provide extensive metadata on questions, students and answers.

ExpMRC

ExpMRC is a benchmark for the Explainability evaluation of Machine Reading Comprehension. ExpMRC contains four subsets of popular MRC datasets with additionally annotated evidences, including SQuAD, CMRC 2018, RACE+ (similar to RACE), and C3, covering span-extraction and multiple-choice questions MRC tasks in both English and Chinese.

M2QA

M2QA (Multi-domain Multilingual Question Answering) is an extractive question answering benchmark for evaluating joint language and domain transfer. M2QA includes 13,500 SQuAD 2.0-style question-answer instances in German, Turkish, and Chinese for the domains of product reviews, news, and creative writing. 40% of the data are unanswerable questions, 60% are answerable.

MilkQA

A question answering dataset from the dairy domain dedicated to the study of consumer questions. The dataset contains 2,657 pairs of questions and answers, written in the Portuguese language and originally collected by the Brazilian Agricultural Research Corporation (Embrapa). All questions were motivated by real situations and written by thousands of authors with very different backgrounds and levels of literacy, while answers were elaborated by specialists from Embrapa's customer service. Source: MilkQA: a Dataset of Consumer Questions for the Task of Answer Selection

MultiReQA

MultiReQA is a cross-domain evaluation for retrieval question answering models. Retrieval question answering (ReQA) is the task of retrieving a sentence-level answer to a question from an open corpus. MultiReQA is a new multi-domain ReQA evaluation suite composed of eight retrieval QA tasks drawn from publicly available QA datasets from the MRQA shared task. MultiReQA contains the sentence boundary annotation from eight publicly available QA datasets including SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, BioASQ, RelationExtraction, and TextbookQA. Five of these datasets, including SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, contain both training and test data, and three, in cluding BioASQ, RelationExtraction, TextbookQA, contain only the test data. Source: https://github.com/google-research-datasets/MultiReQA

ParaMAWPS

This repository contains the code, data, and models of the paper titled "Math Word Problem Solving by Generating Linguistic Variants of Problem Statements" published in the Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop). The work is outlined in a more detailed and expository manner in our Bachelor of Science (B.Sc.) thesis titled "Variational Mathematical Reasoning: Enhancing Math Word Problem Solvers with Linguistic Variants and Disentangled Attention" which can be accessed from the Islamic University of Technology (IUT) Institutional Repository. arXiv anthology GoogleScholar ResearchGate HuggingFace PDF Slides Video Poster Thesis License: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license ## Dataset In order to download our dataset PᴀʀᴀMAWPS, please navigate to the ParaMAWPS folder. We use an $80:10:10$ train-validation-test split for our PᴀʀᴀMAWPS dataset. The splits are available in .json format in the aforementioned folder. ### Data Format Each row consists of a Math Word Problem (MWP). The table below describes what each column signifies. Column Title | Description ------------ | ------------- id | The unique identification number of the sample. Seed problems have id size of $\leq 4$, whereas, variant problems have id size of $> 4$. The last variant of a seed problem (generally with the id "$16000i$", where $i$ is the id of the seed problem) is the inverse variant of the seed problem. original_text | The problem statement of the MWP. The seed problems have the same problem statement as present in the Mᴀᴡᴘs dataset. equation | The equation with a variable \(x\) which solves the MWP quantity_tagged_text | The problem statement of the MWP, where each quantity is replaced with a unique tag \([Q_i]\) quantity_tagged_equation | The equation with a variable \(x\) which solves the MWP, but each quantity is replaced with its unique tag \([Q_i]\) in the problem statement have_constant | Whether the use of a constant value is required to solve the MWP <br> For an MWP sample \(i\) with have_constant label \(C_i\), the boolean label is,<br>$C_i =\begin{cases} \text{FALSE}, & \text{if $i$ requires $0$ constant values}\\\ \text{TRUE}, & \text{if $i$ requires $\geq 1$ constant values}\end{cases}$ ## Types of Variations <img src="https://github.com/Starscream-11813/Variational-Mathematical-Reasoning/raw/main/images/ACLMWP_variationtypes.png" alt="drawing" style="width:1000px;"/> ## Dataset Statistics <img src="https://github.com/Starscream-11813/Variational-Mathematical-Reasoning/raw/main/images/ACLMWP_datasetcomparisontable.png" alt="drawing" style="width:500px;"/> <img src="https://github.com/Starscream-11813/Variational-Mathematical-Reasoning/raw/main/images/ACLMWP_datasetcomparisongraph.png" alt="drawing" style="width:500px;"/> # Methodology <img src="https://github.com/Starscream-11813/Variational-Mathematical-Reasoning/raw/main/images/ACLMWP_architecture2.png" alt="drawing" style="width:1000px;"/> # Results To reproduce the results, please refer to the documentation of MWPToolkit created by Yihuai Lan et al. <img src="https://github.com/Starscream-11813/Variational-Mathematical-Reasoning/raw/main/images/ACLMWP_results.png" alt="drawing" style="width:500px;"/> <img src="https://github.com/Starscream-11813/Variational-Mathematical-Reasoning/raw/main/images/ACLMWP_ablation.png" alt="drawing" style="width:500px;"/> ## Citation If you find this work useful, please cite our paper: bib @inproceedings{raiyan-etal-2023-math, title = "Math Word Problem Solving by Generating Linguistic Variants of Problem Statements", author = "Raiyan, Syed Rifat and Faiyaz, Md Nafis and Kabir, Shah Md. Jawad and Kabir, Mohsinul and Mahmud, Hasan and Hasan, Md Kamrul", booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)", month = jul, year = "2023", address = "Toronto, Canada", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2023.acl-srw.49", doi = "10.18653/v1/2023.acl-srw.49", pages = "362--378", abstract = "The art of mathematical reasoning stands as a fundamental pillar of intellectual progress and is a central catalyst in cultivating human ingenuity. Researchers have recently published a plethora of works centered around the task of solving Math Word Problems (MWP) {---} a crucial stride towards general AI. These existing models are susceptible to dependency on shallow heuristics and spurious correlations to derive the solution expressions. In order to ameliorate this issue, in this paper, we propose a framework for MWP solvers based on the generation of linguistic variants of the problem text. The approach involves solving each of the variant problems and electing the predicted expression with the majority of the votes. We use DeBERTa (Decoding-enhanced BERT with disentangled attention) as the encoder to leverage its rich textual representations and enhanced mask decoder to construct the solution expressions. Furthermore, we introduce a challenging dataset, ParaMAWPS, consisting of paraphrased, adversarial, and inverse variants of selectively sampled MWPs from the benchmark Mawps dataset. We extensively experiment on this dataset along with other benchmark datasets using some baseline MWP solver models. We show that training on linguistic variants of problem statements and voting on candidate predictions improve the mathematical reasoning and robustness of the model. We make our code and data publicly available.", } You can also cite our thesis: bib @phdthesis{raiyan2023variational, type={Bachelor's Thesis}, title={Variational Mathematical Reasoning: Enhancing Math Word Problem Solvers with Linguistic Variants and Disentangled Attention}, author={Raiyan, Syed Rifat and Faiyaz, Md Nafis and Kabir, Shah Md Jawad}, year={2023}, school={Department of Computer Science and Engineering (CSE), Islamic University of Technology}, address={Board Bazar, Gazipur-1704, Dhaka, Bangladesh}, note={Available at \url{http://103.82.172.44:8080/xmlui/handle/123456789/2092}} }

PerCQA

PerCQA is the first Persian dataset for CQA (Community Question Answering). This dataset contains the questions and answers crawled from the most well-known Persian forum.

QUITE

QUITE (Quantifying Uncertainty in natural language Text) is an entirely new benchmark that allows for assessing the capabilities of neural language model-based systems w.r.t. to Bayesian reasoning on a large set of input text that describes probabilistic relationships in natural language text.

ReviewQA

ReviewQA is a question-answering dataset based on hotel reviews. The questions of this dataset are linked to a set of relational understanding competencies that a model is expected to master. Indeed, each question comes with an associated type that characterizes the required competency. Source: ReviewQA: a relational aspect-based opinion reading dataset

RuOpenBookQA

RuOpenBookQA is a QA dataset with multiple-choice elementary-level science questions which probe the understanding of core science facts. Motivation RuOpenBookQA is mainly based on the work of (Mihaylov et al., 2018): it is a QA dataset with multiple-choice elementary-level science questions, which probe the understanding of 1k+ core science facts. Very similar to the pipeline of the RuWorldTree, the dataset includes a corpus of factoids, factoid questions and correct answer. Only one fact is enough to find the correct answer, so this task can be considered easier. { 'ID': '7-674', 'question': 'If a person walks in the direction opposite to the compass needle, they are going (A) west (B) north (C) east (D) south', 'answer': 'D', 'episode': [11], 'perturbation': 'ru_openbook' } Data Fields - ID: a string containing a unique question id - question: a string containing question text with answer options - answer: a string containing the correct answer key (A, B, C or D) - perturbation: a string containing the name of the perturbation applied to text. If no perturbation was applied, the dataset name is used - episode: a list of episodes in which the instance is used. Only used for the train set Data Splits The dataset consists of a training set with labeled examples and a test set in two configurations: - raw data: includes the original data with no additional sampling - episodes: data is split into evaluation episodes and includes several perturbations of test for robustness evaluation Test Perturbations Each training episode in the dataset corresponds to seven test variations, including the original test data and six adversarial test sets, acquired through the modification of the original test through the following text perturbations: - ButterFingers: randomly adds noise to data by mimicking spelling mistakes made by humans through character swaps based on their keyboard distance - Emojify: replaces the input words with the corresponding emojis, preserving their original meaning - EDAdelete: randomly deletes tokens in the text - EDAswap: randomly swaps tokens in the text - BackTranslation: generates variations of the context through back-translation (ru -> en -> ru) - AddSent: replaces one or more choice options with a generated one

Shortcut QA

Recent applications of LLMs in Machine Reading Comprehension (MRC) systems have shown impressive results, but the use of shortcuts, mechanisms triggered by features spuriously correlated to the true label, has emerged as a potential threat to their reliability. We analyze the problem from two angles: LLMs as editors, guided to edit text to mislead LLMs; and LLMs as readers, who answer questions based on the edited text. We introduce a framework that guides an editor to add potential shortcuts-triggers to samples. Using GPT4 as the editor, we find it can successfully edit trigger shortcut in samples that fool LLMs. Analysing LLMs as readers, we observe that even capable LLMs can be deceived using shortcut knowledge. Strikingly, we discover that GPT4 can be deceived by its own edits (15% drop in F1). Our findings highlight inherent vulnerabilities of LLMs to shortcut manipulations. We publish ShortcutQA, a curated dataset generated by our framework for future research.

SimpleDBpediaQA

A new benchmark dataset for simple question answering over knowledge graphs that was created by mapping SimpleQuestions entities and predicates from Freebase to DBpedia. Source: Farewell Freebase: Migrating the SimpleQuestions Dataset to DBpedia

Stanford Schema2QA Dataset

Schema2QA is the first large question answering dataset over real-world Schema.org data. It covers 6 common domains: restaurants, hotels, people, movies, books, and music, based on crawled Schema.org metadata from 6 different websites (Yelp, Hyatt, LinkedIn, IMDb, Goodreads, and last.fm.). In total, there are over 2,000,000 examples for training, consisting of both augmented human paraphrase data and high-quality synthetic data generated by Genie. All questions are annotated with executable virtual assistant programming language ThingTalk. Schema2QA includes challenging evaluation questions collected from crowd workers. Workers are prompted with only what the domain is and what properties are supported. Thus, the sentences are natural and diverse. They also contain entities unseen during training. The collected sentences are manually annotated with ThingTalk by the authors. In total there are over 5,000 examples for dev and test. An example of an evaluation question and its ThingTalk annotation is shown below: "What are the highest ranked burger joints in the 40 mile area around Asheville NC?" sort(aggregateRating.ratingValue desc of @org.schema.Restaurant.Restaurant() filter distance(geo, new Location("asheville nc" )) <= 40 mi && servesCuisine =~ "burger")[1] ;

TempQA-WD

TempQA-WD is a benchmark dataset for temporal reasoning designed to encourage research in extending the present approaches to target a more challenging set of complex reasoning tasks. Specifically, the benchmark is a temporal question answering dataset with the following advantages: (a) it is based on Wikidata, which is the most frequently curated, openly available knowledge base, (b) it includes intermediate sparql queries to facilitate the evaluation of semantic parsing based approaches for KBQA, and (c) it generalizes to multiple knowledge bases: Freebase and Wikidata.

TIME

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

Visual Beliefs

Visual Beliefs is a dataset of abstract scenes to study visual beliefs. The dataset consists of 8-frame scenes, and in each scene a person has a mistaken belief. The dataset can be used for two tasks: predicting who is mistaken and predicting when are they mistaken. Source: Who is Mistaken?

VizWiz-Priv

VizWiz-Priv includes 8,862 regions showing private content across 5,537 images taken by blind people. Of these, 1,403 are paired with questions and 62% of those directly ask about the private content. Source: VizWiz-Priv: A Dataset for Recognizing the Presence and Purpose of Private Visual Information in Images Taken by Blind People

VizWiz-QualityIssues

A large-scale dataset that links the assessment of image quality issues to two practical vision tasks: image captioning and visual question answering. Source: Assessing Image Quality Issues for Real-World Problems

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.