Papers with Code 아카이브

Question Answering

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
19
평가표
153
보존된 평가 기록
1,781

Question answering can be segmented into domain-specific tasks like community question answering and knowledge-base question answering. Popular benchmark datasets for evaluation question answering systems include SQuAD, HotPotQA, bAbI, TriviaQA, WikiQA, and many others. Models for question answering are typically evaluated on metrics like EM and F1. Some recent top performing models are T5 and XLNet.

( Image credit: SQuAD )

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

MapEval-Textual

MapEval-Textual contains 300 context-question-answer triplets. The necessary geo-spatial information is provided in the context. The task is to answer question based on the factual data provided in the context.

MathMC

Existing arithmetic benchmarks have a limited number of multiple-choice questions. To address this gap, MathMC is created including 1,000 Chinese mathematical multiple-choice questions with detailed explanations and focusing on math problems typically encountered in grades 4 to 6. It features a wide range of question types, including arithmetic, algebra, geometry, statistics, reasoning, and more, enhancing the diversity of current Chinese arithmetic datasets.

MathToF

Existing arithmetic benchmarks have a limited number of True-or-False questions. To address this gap, MathToF is created including 1,000 Chinese mathematical True-or-False questions with detailed explanations and focusing on math problems typically encountered in grades 4 to 6. It features a wide range of question types, including arithmetic, algebra, geometry, statistics, reasoning, and more, enhancing the diversity of current Chinese arithmetic datasets.

MatTools

pymatgen_code_qa benchmark: qa_benchmark/generated_qa/generation_results_code.json, which consists of 34,621 QA pairs. pymatgen_code_doc benchmark: qa_benchmark/generated_qa/generation_results_doc.json, which consists of 34,604 QA pairs. real-world tool-usage benchmark: src/question_segments, which consists of 49 questions (138 tasks). One subfolder means a question with problem statement, property list and verification code.

MF3QA

real-world doctor-patient question- answering dataset cleaned manually and automatically

MILU

Overview MILU (Multi-task Indic Language Understanding Benchmark) is a comprehensive evaluation dataset designed to assess the performance of Large Language Models (LLMs) across 11 Indic languages. It spans 8 domains and 42 subjects, reflecting both general and culturally specific knowledge from India. ## Key Features - Languages: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu, and English - Domains: 8 diverse domains including Arts & Humanities, Social Sciences, STEM, and more - Subjects: 42 subjects covering a wide range of topics - Questions: ~85,000 multiple-choice questions - Cultural Relevance: Incorporates India-specific knowledge from regional and state-level examinations ## Dataset Statistics | Language | Total Questions | Translated Questions | Avg Words Per Question | |----------|-----------------|----------------------|------------------------| | Bengali | 7138 | 1601 | 15.72 | | Gujarati | 5327 | 2755 | 16.69 | | Hindi | 15450 | 115 | 20.63 | | Kannada | 6734 | 1522 | 12.83 | | Malayalam| 4670 | 1534 | 12.82 | | Marathi | 7424 | 1235 | 18.8 | | Odia | 5025 | 1452 | 15.63 | | Punjabi | 4363 | 2341 | 19.9 | | Tamil | 7059 | 1524 | 13.32 | | Telugu | 7847 | 1298 | 16.13 | | English | 14036 | - | 22.01 | | Total| 85073 | 15377 | 16.77 (avg) | ## Dataset Structure ### Test Set The test set consists of the MILU (Multi-task Indic Language Understanding) benchmark, which contains approximately 85,000 multiple-choice questions across 11 Indic languages. ### Validation Set The dataset includes a separate validation set of 9,157 samples that can be used for few-shot examples during evaluation. This validation set was created by sampling from each of the 42 subject tags, which were then condensed into 8 broader domains. This approach ensures a balanced representation across subjects and domains, allowing for consistent few-shot prompting across different models and experiments. ### Subjects spanning MILU | Domain | Subjects | |--------|----------| | Arts & Humanities | Architecture and Design, Arts and Culture, Education, History, Language Studies, Literature and Linguistics, Media and Communication, Music and Performing Arts, Religion and Spirituality | | Business Studies | Business and Management, Economics, Finance and Investment | | Engineering & Tech | Energy and Power, Engineering, Information Technology, Materials Science, Technology and Innovation, Transportation and Logistics | | Environmental Sciences | Agriculture, Earth Sciences, Environmental Science, Geography | | Health & Medicine | Food Science, Health and Medicine | | Law & Governance | Defense and Security, Ethics and Human Rights, Law and Ethics, Politics and Governance | | Math and Sciences | Astronomy and Astrophysics, Biology, Chemistry, Computer Science, Logical Reasoning, Mathematics, Physics | | Social Sciences | Anthropology, International Relations, Psychology, Public Administration, Social Welfare and Development, Sociology, Sports and Recreation | ## Usage Since this is a gated dataset, after your request for accessing the dataset is accepted, you can set your HuggingFace token: bash export HF_TOKEN=YOUR_TOKEN_HERE To load the MILU dataset for a Language: python from datasets import load_dataset language = 'Hindi' # Use 'test' split for evaluation & 'validation' split for few-shot split = 'test' language_data = load_dataset("ai4bharat/MILU", data_dir=language, split=split, token=True) print(language_data[0]) ## Evaluation We evaluated 45 different LLMs on MILU, including: - Closed proprietary models (e.g., GPT-4o, Gemini-1.5) - Open-source multilingual models - Language-specific fine-tuned models Key findings: - GPT-4o achieved the highest average accuracy at 72% - Open multilingual models outperformed language-specific fine-tuned models - Models performed better in high-resource languages compared to low-resource ones - Performance was lower in culturally relevant areas (e.g., Arts & Humanities) compared to general fields like STEM For detailed results and analysis, please refer to our paper. ## Citation If you use MILU in your research, please cite our paper: @misc{verma2024milumultitaskindiclanguage, title={MILU: A Multi-task Indic Language Understanding Benchmark}, author={Sshubam Verma and Mohammed Safi Ur Rahman Khan and Vishwajeet Kumar and Rudra Murthy and Jaydeep Sen}, year={2024}, eprint={2411.02538}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2411.02538}, } ## License This dataset is released under the MIT License. ## Contact For any questions or feedback, please contact: - Sshubam Verma (sshubamverma@ai4bharat.org) - Mohammed Safi Ur Rahman Khan (safikhan@ai4bharat.org) - Rudra Murthy (rmurthyv@in.ibm.com) - Vishwajeet Kumar (vishk024@in.ibm.com) ## Links - GitHub Repository - Paper - Hugging Face Dataset

MLQuestions

MLQuestions is a domain-adaptation dataset for the machine learning domain containing 50K unaligned passages and 35K unaligned questions, and 3K aligned passage and question pairs.

MMInstruct-GPT4V

Vision-language supervised fine-tuning effectively enhances VLLM performance, but existing visual instruction tuning datasets have limitations: 1. Instruction Annotation Quality: Despite strong performance, advanced VLLMs may generate instructions with inaccuracies, such as hallucinations. 2. Instruction and Image Diversity: Limited instruction types and lack of diverse image data impact the model's ability to generate varied and realistic outputs. MMInstruct Dataset To address these challenges, we created the MMInstruct dataset, featuring: - 973K instructions from 24 domains - Four instruction types: Judgement, Multiple-Choice, Long Visual Question Answering, and Short Visual Question Answering.

MMOS

Mix of Minimal Optimal Sets (MMOS) of dataset has two advantages for two aspects, higher performance and lower construction costs on math reasoning.

NitiBench

A benchmark for legal question answering. The data only contains test set and got two splits: NitiBench-CCL representing Thai corporate and commercial law, and NitiBench-Tax containing official tax ruling in Thai scraped from official Revenue Department website.

ORKG-QA

A preliminary dataset of related tables and a corresponding set of natural language questions. Source: Question Answering on Scholarly Knowledge Graphs

Pentachromatic Cultural Palette Dataset

Pentachromatic Cultural Palette Dataset is characterized by unique cultural semantics and values. It is constructed through a carefully - designed multi - step data synthesis process based on PRISM dataset, focusing on the cultural perspectives of different continents (Africa, Asia, Europe, America, and Oceania).

Persian NLP Hub

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

PersianQA

PersianQA: a dataset for Persian Question Answering Persian Question Answering (PersianQA) Dataset is a reading comprehension dataset on Persian Wikipedia. The crowd-sourced the dataset consists of more than 9,000 entries. Each entry can be either an impossible-to-answer or a question with one or more answers spanning in the passage (the context) from which the questioner proposed the question. Much like the SQuAD2.0 dataset, the impossible or unanswerable questions can be utilized to create a system which "knows that it doesn't know the answer". Moreover, the dataset has 900 test data available. On top of that, the very first models trained on the dataset, Transformers, are available online. All the crowd workers of the dataset are native Persian speakers. Also, it worth mentioning that the contexts are collected from all categories of the Wiki (Historical, Religious, Geography, Science, etc).

Phrase-in-Context

Phrase in Context is a curated benchmark for phrase understanding and semantic search, consisting of three tasks of increasing difficulty: Phrase Similarity (PS), Phrase Retrieval (PR) and Phrase Sense Disambiguation (PSD). The datasets are annotated by 13 linguistic experts on Upwork and verified by two groups: ~1000 AMT crowdworkers and another set of 5 linguistic experts. PiC benchmark is distributed under CC-BY-NC 4.0.

Pirá

A large set of questions and answers about the ocean and the Brazilian coast both in Portuguese and English. Pirá is a crowdsourced question answering (QA) dataset on the ocean and the Brazilian coast designed for reading comprehension. The dataset contains 2261 QA sets, as well as the texts associated with them. Each QA set contains at least four elements: a question in Portuguese and in English, and an answer in Portuguese and in English. Around 90% of the QA sets also contain human evaluations. Pirá is, to the best of our knowledge, the first QA dataset with supporting texts in Portuguese, and, perhaps more importantly, the first bilingual QA dataset that includes Portuguese as one of its languages. Pirá is also the first QA dataset in Portuguese with unanswerable questions so as to allow the study of answer triggering. Finally, it is the first QA dataset that tackles scientific knowledge about the ocean, climate change, and marine biodiversity.

PQ-decaNLP

Multitask learning has led to significant advances in Natural Language Processing, including the decaNLP benchmark where question answering is used to frame 10 natural language understanding tasks in a single model. PQ-decaNLP is a crowd-sourced corpus of paraphrased questions, annotated with paraphrase phenomena. This enables analysis of how transformations such as swapping the class labels and changing the sentence modality lead to a large performance degradation.

PQAref

The PQAref dataset is a dataset for fine-tuning large language models for referenced question-answering in biomedical domain. The dataset contains 3 components: Instruction - question that is supposed to be answered Abstracts - set of 10 relevant abstracts retrieved from PubMed by an IR system. They contain the PubMed id, abstract title and the content of the abstract Answer - expected answer, with references in the form of PubMed IDs. The dataset was created semi-automatically, utilizing questions available from PubMedQA dataset. The dataset contains 9,075 samples, split into training, validation and test set in proportion 80%:10%:10%.

QASiNa

Question Answering Sirah Nabawiyah (QASiNa) Dataset is a reading comprehension dataset consists of QA from Sirah Nabawiyah literature in Indonesian Language

QASports

Sport is one of the most popular and revenue-generating forms of entertainment. Therefore, analyzing data related to this domain introduces several opportunities for Question Answering (QA) systems, such as supporting tactical decision-making. But, to develop and evaluate QA systems, researchers and developers need datasets that contain questions and their corresponding answers. In this paper, we focus on this issue. We propose QASports, the first large sports question answering dataset for extractive answer questions. QASports contains more than 1.5 million triples of questions, answers, and context about three popular sports: soccer, American football, and basketball. We describe the QASports processes of data collection and questions and answers generation. We also describe the characteristics of the QASports data. Furthermore, we analyze the sources used to obtain raw data and investigate the usability of QASports by issuing "wh-queries". Moreover, we describe scenarios for using QASports, highlighting its importance for training and evaluating QA systems.

Reddit Ideology Database

Dataset with articles posted in the r/Liberal and r/Conservative subreddits. In total, we collected a corpus of 226,010 articles. We have collected news articles to understand political expression through the shared news articles.

RoomSpace

RoomSpace: a new benchmark designed to evaluate language models on spatial reasoning tasks demanding spatial relation knowledge and multi-hop reasoning. RoomSpace encompasses a comprehensive range of qualitative spatial relationships, including topological, directional, and distance relations. These relationships are presented from various viewpoints, with differing levels of granularity and density of relational constraints to simulate real-world complexities. This approach promotes a more accurate assessment of language models' capabilities in spatial reasoning tasks.

SchizzoSQUAD

The “Mental Health” forum was used, a forum dedicated to people suffering from schizophrenia and different mental disorders. Relevant posts of active users, who regularly participate, were extrapolated providing a new method of obtaining low-bias content and without privacy issues. This corpus i then processed to offer a SQUAD (Standford Question Answering Dataset) version in order to train a ML QA model. The structure is a Json with different keys with the following structure: { "data": [ { "paragraphs": [ { "qas": [ { "question": "What is the patient afraid of?", "id": 304425, "answers": [ { "answer_id": 305544, "document_id": 501182, "question_id": 304425, "text": "I’m afraid of people except for my family and a few close friends", "answer_start": 29769, "answer_category": null } ], "is_impossible": false } ], "context": "\t" I don’t understand why people send aid to other countries and go on missions when our own country people are suffering and have nothing, in the us at least."\n\t"7-21-19 Said \"What is she afraid of? Afraid of family!?, "document_id": 501182 } ] }

ScienceExamCER

ScienceExamCER is a collection of resources for studying explanation-centered inference, including explanation graphs for 1,680 questions, with 4,950 tablestore rows, and other analyses of the knowledge required to answer elementary and middle-school science questions. Source: Allen Institute for AI

SCIMAT

SCIMAT is a large question-answer dataset for mathematics and science problems; such dataset can have impact on online education, intelligent tutoring and automated grading.

SE-PQA

Personalization in Information Retrieval is a topic studied for a long time. Nevertheless, there is still a lack of high-quality, real-world datasets to conduct large-scale experiments and evaluate models for personalized search. This paper contributes to fill this gap by introducing SE-PQA (StackExchange - Personalized Question Answering), a new resource to design and evaluate personalized models related to the two tasks of community Question Answering (cQA). The contributed dataset includes more than 1 million queries and 2 million answers, annotated with a rich set of features modeling the social interactions among the users of a popular cQA platform. We describe the characteristics of SE-PQA and detail the features associated with both questions and answers. We also provide reproducible baseline methods for the cQA task based on the resource, including deep learning models and personalization approaches. The results of the preliminary experiments conducted show the appropriateness of SE-PQA to train effective cQA models; they also show that personalization improves remarkably the effectiveness of all the methods tested. Furthermore, we show the benefits in terms of robustness and generalization of combining data from multiple communities for personalization purposes.

SF20K

Short-Films 20K (SF20K) is the largest publicly available movie dataset. SF20K is composed of 20,143 amateur films and offers long-term video tasks in the form of multiple-choice and open-ended question answering.

simply-CLEVR

The simply-CLEVR dataset aims to provide a benchmark dataset that can be used for transparent quantitative evaluation of explanation methods (aka heatmaps/XAI methods). It is made of simple Visual Question Answering (VQA) questions, which are derived from the original CLEVR task, and where each question is accompanied by two Ground Truth Masks that serve as a basis for evaluating explanations on the input image. Source: https://github.com/ahmedmagdiosman/simply-clevr-dataset Image Source: https://github.com/ahmedmagdiosman/simply-clevr-dataset

SOTU_QA_2023

Curated QA Benchmark on State of the Union Address 2023. It contains curated question and answers based on knowledge presented in State of the Union Address 2023 (in Feb). It is especially useful for tool-augmented LMs / ALMs to examine the model's ability in answering over private document.

TimelineKGQA

Question answering over temporal knowledge graphs (TKGs) is crucial for understanding evolving facts and relationships, yet its development is hindered by limited datasets and difficulties in generating custom QA pairs. We propose a novel categorization framework based on timeline-context relationships, along with \textbf{TimelineKGQA}, a universal temporal QA generator applicable to any TKGs. The code is available at: \url{https://github.com/PascalSun/TimelineKGQA} as an open source Python package. We present a comprehensive temporal question categorization framework and a universal TKGQA dataset generator that addresses the limitations of existing datasets. Our framework introduces a systematic approach to classify question complexity through context facts, answer focus, and temporal operations, while identifying four key temporal capabilities (TCR, TPR, TSO, and TAO). Using our \textbf{TimelineKGQA} generator to create two benchmark datasets, we demonstrate that question difficulty aligns with our complexity categorization through empirical evaluation. This work provides a foundation for developing and evaluating more advanced TKGQA solutions, and enables widespread application in private domains.

TinySocial

TinySocial is a dataset to enable research on Social Visual Question Answering. Source: Characterizing Datasets for Social Visual Question Answering, and the New TinySocial Dataset

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.