MapEval-Textual
MapEval-Textual contains 300 context-question-answer triplets. The necessary geo-spatial information is provided in the context. The task is to answer question based on the factual data provided in the context.
태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.
Question answering can be segmented into domain-specific tasks like community question answering and knowledge-base question answering. Popular benchmark datasets for evaluation question answering systems include SQuAD, HotPotQA, bAbI, TriviaQA, WikiQA, and many others. Models for question answering are typically evaluated on metrics like EM and F1. Some recent top performing models are T5 and XLNet.
( Image credit: SQuAD )
출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.
MapEval-Textual contains 300 context-question-answer triplets. The necessary geo-spatial information is provided in the context. The task is to answer question based on the factual data provided in the context.
Existing arithmetic benchmarks have a limited number of multiple-choice questions. To address this gap, MathMC is created including 1,000 Chinese mathematical multiple-choice questions with detailed explanations and focusing on math problems typically encountered in grades 4 to 6. It features a wide range of question types, including arithmetic, algebra, geometry, statistics, reasoning, and more, enhancing the diversity of current Chinese arithmetic datasets.
Existing arithmetic benchmarks have a limited number of True-or-False questions. To address this gap, MathToF is created including 1,000 Chinese mathematical True-or-False questions with detailed explanations and focusing on math problems typically encountered in grades 4 to 6. It features a wide range of question types, including arithmetic, algebra, geometry, statistics, reasoning, and more, enhancing the diversity of current Chinese arithmetic datasets.
pymatgen_code_qa benchmark: qa_benchmark/generated_qa/generation_results_code.json, which consists of 34,621 QA pairs. pymatgen_code_doc benchmark: qa_benchmark/generated_qa/generation_results_doc.json, which consists of 34,604 QA pairs. real-world tool-usage benchmark: src/question_segments, which consists of 49 questions (138 tasks). One subfolder means a question with problem statement, property list and verification code.
A comprehensive Turkish dataset for question-answering tasks in medical domain
real-world doctor-patient question- answering dataset cleaned manually and automatically
real-world doctor-patient question- answering dataset
bash export HF_TOKEN=YOUR_TOKEN_HERE To load the MILU dataset for a Language: python from datasets import load_dataset language = 'Hindi' # Use 'test' split for evaluation & 'validation' split for few-shot split = 'test' language_data = load_dataset("ai4bharat/MILU", data_dir=language, split=split, token=True) print(language_data[0]) ## Evaluation We evaluated 45 different LLMs on MILU, including: - Closed proprietary models (e.g., GPT-4o, Gemini-1.5) - Open-source multilingual models - Language-specific fine-tuned models Key findings: - GPT-4o achieved the highest average accuracy at 72% - Open multilingual models outperformed language-specific fine-tuned models - Models performed better in high-resource languages compared to low-resource ones - Performance was lower in culturally relevant areas (e.g., Arts & Humanities) compared to general fields like STEM For detailed results and analysis, please refer to our paper. ## Citation If you use MILU in your research, please cite our paper: @misc{verma2024milumultitaskindiclanguage, title={MILU: A Multi-task Indic Language Understanding Benchmark}, author={Sshubam Verma and Mohammed Safi Ur Rahman Khan and Vishwajeet Kumar and Rudra Murthy and Jaydeep Sen}, year={2024}, eprint={2411.02538}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2411.02538}, } ## License This dataset is released under the MIT License. ## Contact For any questions or feedback, please contact: - Sshubam Verma (sshubamverma@ai4bharat.org) - Mohammed Safi Ur Rahman Khan (safikhan@ai4bharat.org) - Rudra Murthy (rmurthyv@in.ibm.com) - Vishwajeet Kumar (vishk024@in.ibm.com) ## Links - GitHub Repository - Paper - Hugging Face DatasetMLQuestions is a domain-adaptation dataset for the machine learning domain containing 50K unaligned passages and 35K unaligned questions, and 3K aligned passage and question pairs.
Vision-language supervised fine-tuning effectively enhances VLLM performance, but existing visual instruction tuning datasets have limitations: 1. Instruction Annotation Quality: Despite strong performance, advanced VLLMs may generate instructions with inaccuracies, such as hallucinations. 2. Instruction and Image Diversity: Limited instruction types and lack of diverse image data impact the model's ability to generate varied and realistic outputs. MMInstruct Dataset To address these challenges, we created the MMInstruct dataset, featuring: - 973K instructions from 24 domains - Four instruction types: Judgement, Multiple-Choice, Long Visual Question Answering, and Short Visual Question Answering.
Mix of Minimal Optimal Sets (MMOS) of dataset has two advantages for two aspects, higher performance and lower construction costs on math reasoning.
A benchmark for legal question answering. The data only contains test set and got two splits: NitiBench-CCL representing Thai corporate and commercial law, and NitiBench-Tax containing official tax ruling in Thai scraped from official Revenue Department website.
NText is an eight million words dataset extracted and preprocessed from nuclear research papers and thesis. Source: NukeBERT: A Pre-trained language model for Low Resource Nuclear Domain
A preliminary dataset of related tables and a corresponding set of natural language questions. Source: Question Answering on Scholarly Knowledge Graphs
Pentachromatic Cultural Palette Dataset is characterized by unique cultural semantics and values. It is constructed through a carefully - designed multi - step data synthesis process based on PRISM dataset, focusing on the cultural perspectives of different continents (Africa, Asia, Europe, America, and Oceania).
Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset
Phrase in Context is a curated benchmark for phrase understanding and semantic search, consisting of three tasks of increasing difficulty: Phrase Similarity (PS), Phrase Retrieval (PR) and Phrase Sense Disambiguation (PSD). The datasets are annotated by 13 linguistic experts on Upwork and verified by two groups: ~1000 AMT crowdworkers and another set of 5 linguistic experts. PiC benchmark is distributed under CC-BY-NC 4.0.
Physical concept understanding benchmark. Source: The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding
A large set of questions and answers about the ocean and the Brazilian coast both in Portuguese and English. Pirá is a crowdsourced question answering (QA) dataset on the ocean and the Brazilian coast designed for reading comprehension. The dataset contains 2261 QA sets, as well as the texts associated with them. Each QA set contains at least four elements: a question in Portuguese and in English, and an answer in Portuguese and in English. Around 90% of the QA sets also contain human evaluations. Pirá is, to the best of our knowledge, the first QA dataset with supporting texts in Portuguese, and, perhaps more importantly, the first bilingual QA dataset that includes Portuguese as one of its languages. Pirá is also the first QA dataset in Portuguese with unanswerable questions so as to allow the study of answer triggering. Finally, it is the first QA dataset that tackles scientific knowledge about the ocean, climate change, and marine biodiversity.
Multitask learning has led to significant advances in Natural Language Processing, including the decaNLP benchmark where question answering is used to frame 10 natural language understanding tasks in a single model. PQ-decaNLP is a crowd-sourced corpus of paraphrased questions, annotated with paraphrase phenomena. This enables analysis of how transformations such as swapping the class labels and changing the sentence modality lead to a large performance degradation.
The PQAref dataset is a dataset for fine-tuning large language models for referenced question-answering in biomedical domain. The dataset contains 3 components: Instruction - question that is supposed to be answered Abstracts - set of 10 relevant abstracts retrieved from PubMed by an IR system. They contain the PubMed id, abstract title and the content of the abstract Answer - expected answer, with references in the form of PubMed IDs. The dataset was created semi-automatically, utilizing questions available from PubMedQA dataset. The dataset contains 9,075 samples, split into training, validation and test set in proportion 80%:10%:10%.
Question Answering Sirah Nabawiyah (QASiNa) Dataset is a reading comprehension dataset consists of QA from Sirah Nabawiyah literature in Indonesian Language
Sport is one of the most popular and revenue-generating forms of entertainment. Therefore, analyzing data related to this domain introduces several opportunities for Question Answering (QA) systems, such as supporting tactical decision-making. But, to develop and evaluate QA systems, researchers and developers need datasets that contain questions and their corresponding answers. In this paper, we focus on this issue. We propose QASports, the first large sports question answering dataset for extractive answer questions. QASports contains more than 1.5 million triples of questions, answers, and context about three popular sports: soccer, American football, and basketball. We describe the QASports processes of data collection and questions and answers generation. We also describe the characteristics of the QASports data. Furthermore, we analyze the sources used to obtain raw data and investigate the usability of QASports by issuing "wh-queries". Moreover, we describe scenarios for using QASports, highlighting its importance for training and evaluating QA systems.
Dataset with articles posted in the r/Liberal and r/Conservative subreddits. In total, we collected a corpus of 226,010 articles. We have collected news articles to understand political expression through the shared news articles.
RGRS is a dataset for collaboratior recommendation on the ResearchGate academic social network. The data has been collected from Jan. 2019 to April 2019 and includes raw data of 3980 RG users. Source: Presenting a Dataset for Collaborator Recommending Systems in Academic Social Network: a Case Study on ReseatchGate
RoomSpace: a new benchmark designed to evaluate language models on spatial reasoning tasks demanding spatial relation knowledge and multi-hop reasoning. RoomSpace encompasses a comprehensive range of qualitative spatial relationships, including topological, directional, and distance relations. These relationships are presented from various viewpoints, with differing levels of granularity and density of relational constraints to simulate real-world complexities. This approach promotes a more accurate assessment of language models' capabilities in spatial reasoning tasks.
The “Mental Health” forum was used, a forum dedicated to people suffering from schizophrenia and different mental disorders. Relevant posts of active users, who regularly participate, were extrapolated providing a new method of obtaining low-bias content and without privacy issues. This corpus i then processed to offer a SQUAD (Standford Question Answering Dataset) version in order to train a ML QA model. The structure is a Json with different keys with the following structure: { "data": [ { "paragraphs": [ { "qas": [ { "question": "What is the patient afraid of?", "id": 304425, "answers": [ { "answer_id": 305544, "document_id": 501182, "question_id": 304425, "text": "I’m afraid of people except for my family and a few close friends", "answer_start": 29769, "answer_category": null } ], "is_impossible": false } ], "context": "\t" I don’t understand why people send aid to other countries and go on missions when our own country people are suffering and have nothing, in the us at least."\n\t"7-21-19 Said \"What is she afraid of? Afraid of family!?, "document_id": 501182 } ] }
ScienceExamCER is a collection of resources for studying explanation-centered inference, including explanation graphs for 1,680 questions, with 4,950 tablestore rows, and other analyses of the knowledge required to answer elementary and middle-school science questions. Source: Allen Institute for AI
SCIMAT is a large question-answer dataset for mathematics and science problems; such dataset can have impact on online education, intelligent tutoring and automated grading.
Personalization in Information Retrieval is a topic studied for a long time. Nevertheless, there is still a lack of high-quality, real-world datasets to conduct large-scale experiments and evaluate models for personalized search. This paper contributes to fill this gap by introducing SE-PQA (StackExchange - Personalized Question Answering), a new resource to design and evaluate personalized models related to the two tasks of community Question Answering (cQA). The contributed dataset includes more than 1 million queries and 2 million answers, annotated with a rich set of features modeling the social interactions among the users of a popular cQA platform. We describe the characteristics of SE-PQA and detail the features associated with both questions and answers. We also provide reproducible baseline methods for the cQA task based on the resource, including deep learning models and personalization approaches. The results of the preliminary experiments conducted show the appropriateness of SE-PQA to train effective cQA models; they also show that personalization improves remarkably the effectiveness of all the methods tested. Furthermore, we show the benefits in terms of robustness and generalization of combining data from multiple communities for personalization purposes.
Short-Films 20K (SF20K) is the largest publicly available movie dataset. SF20K is composed of 20,143 amateur films and offers long-term video tasks in the form of multiple-choice and open-ended question answering.
The simply-CLEVR dataset aims to provide a benchmark dataset that can be used for transparent quantitative evaluation of explanation methods (aka heatmaps/XAI methods). It is made of simple Visual Question Answering (VQA) questions, which are derived from the original CLEVR task, and where each question is accompanied by two Ground Truth Masks that serve as a basis for evaluating explanations on the input image. Source: https://github.com/ahmedmagdiosman/simply-clevr-dataset Image Source: https://github.com/ahmedmagdiosman/simply-clevr-dataset
Curated QA Benchmark on State of the Union Address 2023. It contains curated question and answers based on knowledge presented in State of the Union Address 2023 (in Feb). It is especially useful for tool-augmented LMs / ALMs to examine the model's ability in answering over private document.
Question answering over temporal knowledge graphs (TKGs) is crucial for understanding evolving facts and relationships, yet its development is hindered by limited datasets and difficulties in generating custom QA pairs. We propose a novel categorization framework based on timeline-context relationships, along with \textbf{TimelineKGQA}, a universal temporal QA generator applicable to any TKGs. The code is available at: \url{https://github.com/PascalSun/TimelineKGQA} as an open source Python package. We present a comprehensive temporal question categorization framework and a universal TKGQA dataset generator that addresses the limitations of existing datasets. Our framework introduces a systematic approach to classify question complexity through context facts, answer focus, and temporal operations, while identifying four key temporal capabilities (TCR, TPR, TSO, and TAO). Using our \textbf{TimelineKGQA} generator to create two benchmark datasets, we demonstrate that question difficulty aligns with our complexity categorization through empirical evaluation. This work provides a foundation for developing and evaluating more advanced TKGQA solutions, and enables widespread application in private domains.
TinySocial is a dataset to enable research on Social Visual Question Answering. Source: Characterizing Datasets for Social Visual Question Answering, and the New TinySocial Dataset
출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.