RAG

We present PeerQA, a real-world, scientific, document-level Question Answering (QA) dataset. PeerQA questions have been sourced from peer reviews, which contain questions that reviewers raised while thoroughly examining the scientific article. Answers have been annotated by the original authors of each paper. The dataset contains 579 QA pairs from 208 academic articles, with a majority from ML and NLP, as well as a subset of other scientific communities like Geoscience and Public Health. PeerQA supports three critical tasks for developing practical QA systems: Evidence retrieval, unanswerable question classification, and answer generation. We provide a detailed analysis of the collected dataset and conduct experiments establishing baseline systems for all three tasks. Our experiments and analyses reveal the need for decontextualization in document-level retrieval, where we find that even simple decontextualization approaches consistently improve retrieval performance across architectures. On answer generation, PeerQA serves as a challenging benchmark for long-context modeling, as the papers have an average size of 12k tokens.

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

CompMix-IR Dataset Overview: Characteristics: CompMix-IR is a heterogeneous knowledge retrieval benchmark dataset, featuring four knowledge types (text, knowledge graphs, tables, and infoboxes), 9,400+ QA pairs, and a corpus of 10 million entries. It supports two retrieval scenarios: retrieving across all knowledge types or retrieving specific types based on user instructions. Motivation: It addresses the limitations of existing benchmarks by providing a more comprehensive and realistic dataset that reflects real-world retrieval needs with diverse knowledge sources and user intents. Potential Use Cases: Ideal for developing and evaluating heterogeneous IR models, instruction-aware retrieval systems, and open-domain QA systems. It can also be used for benchmarking, cross-domain IR research, and enhancing the adaptability and robustness of retrieval models.

The Official dataset proposed int the paper Context Awareness Gate For Retrieval Augmented Generation

Open-source dataset

PubMedQA-MetaGen: Metadata-Enriched PubMedQA Corpus Dataset Summary PubMedQA-MetaGen is a metadata-enriched version of the PubMedQA biomedical question-answering dataset, created using the MetaGenBlendedRAG enrichment pipeline. The dataset contains both the original and enriched versions of the corpus, enabling direct benchmarking of retrieval-augmented and semantic search approaches in biomedical NLP. Files Provided PubMedQA_original_corpus.json This file contains the original PubMedQA corpus, formatted directly from the official PubMedQA dataset. Each record includes the biomedical question, context (abstract), and answer fields, mirroring the original dataset structure. PubMedQA_corpus_with_metadata.json This file contains the metadata-enriched version, created by processing the original corpus through the MetaGenBlendedRAG pipeline. In addition to the original fields, each entry is augmented with structured metadata—including key concepts, MeSH terms, automatically generated keywords, extracted entities, and LLM-generated summaries—designed to support advanced retrieval and RAG research. How to Use RAG evaluation: Benchmark your retrieval-augmented QA models using the enriched context for higher recall and precision. Semantic Search: Build improved biomedical search engines leveraging topic, entity, and keyword metadata. NLP & LLM Fine-tuning: Use for fine-tuning models that benefit from structured biomedical context. Dataset Structure Each sample contains: Original fields: Question, context (abstract), answer Enriched fields (in PubMedQA_corpus_with_metadata.json only): Key concepts and topics Extracted MeSH terms and UMLS entities Automatically generated keywords Section/type labels LLM-generated summaries/metadata Document identifiers and links Dataset Creation Process Source: Original PubMedQA dataset. Metadata Enrichment: Applied the MetaGenBlendedRAG pipeline (rule-based, NLP, and LLM-driven enrichment). Outputs: Two files—original and enriched—supporting both traditional and metadata-driven research. Intended Use and Limitations For research and educational use in biomedical QA, RAG, semantic retrieval, and metadata enrichment evaluation. Note: Some metadata fields generated by LLMs may vary in quality; users should verify outputs for critical applications.

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset