Text Retrieval is the task of finding the most text result (such as an answer, paragraph, or passage) given a query (which could be a question, keywords, or any relevant text)
데이터셋
출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.
The Natural Questions corpus is a question answering dataset containing 307,373 training examples, 7,830 development examples, and 7,842 test examples. Each example is comprised of a google.com query and a corresponding Wikipedia page. Each Wikipedia page has a passage (or long answer) annotated on the page that answers the question and one or more short spans from the annotated passage containing the actual answer. The long and the short answer annotations can however be empty. If they are both empty, then there is no answer on the page at all. If the long answer annotation is non-empty, but the short answer annotation is empty, then the annotated passage answers the question but no explicit short answer could be found. Finally 1% of the documents have a passage annotated with a short answer that is “yes” or “no”, instead of a list of short spans. Source: A BERT Baseline for the Natural Questions Image Source: https://paperswithcode.com/paper/natural-questions-a-benchmark-for-question/
The MS MARCO (Microsoft MAchine Reading Comprehension) is a collection of datasets focused on deep learning in search. The first dataset was a question answering dataset featuring 100,000 real Bing questions and a human generated answer. Over time the collection was extended with a 1,000,000 question dataset, a natural language generation dataset, a passage ranking dataset, keyphrase extraction dataset, crawling dataset, and a conversational search. Source: https://microsoft.github.io/msmarco/ Image Source: https://arxiv.org/pdf/1809.08267.pdf
HotpotQA is a question answering dataset collected on the English Wikipedia, containing about 113K crowd-sourced questions that are constructed to require the introduction paragraphs of two Wikipedia articles to answer. Each question in the dataset comes with the two gold paragraphs, as well as a list of sentences in these paragraphs that crowdworkers identify as supporting facts necessary to answer the question. A diverse range of reasoning strategies are featured in HotpotQA, including questions involving missing entities in the question, intersection questions (What satisfies property A and property B?), and comparison questions, where two entities are compared by a common attribute, among others. In the few-document distractor setting, the QA models are given ten paragraphs in which the gold paragraphs are guaranteed to be found; in the open-domain fullwiki setting, the models are only given the question and the entire Wikipedia. Models are evaluated on their answer accuracy and explainability, where the former is measured as overlap between the predicted and gold answers with exact match (EM) and unigram F1, and the latter concerns how well the predicted supporting fact sentences match human annotation (Supporting Fact EM/F1). A joint metric is also reported on this dataset, which encourages systems to perform well on both tasks simultaneously. Source: Answering Complex Open-domain Questions Through Iterative Query Generation Image Source: Yang et al
DBpedia (from "DB" for "database") is a project aiming to extract structured content from the information created in the Wikipedia project. DBpedia allows users to semantically query relationships and properties of Wikipedia resources, including links to other related datasets. Source: https://en.wikipedia.org/wiki/DBpedia
FEVER is a publicly available dataset for fact extraction and verification against textual sources. It consists of 185,445 claims manually verified against the introductory sections of Wikipedia pages and classified as SUPPORTED, REFUTED or NOTENOUGHINFO. For the first two classes, systems and annotators need to also return the combination of sentences forming the necessary evidence supporting or refuting the claim. The claims were generated by human annotators extracting claims from Wikipedia and mutating them in a variety of ways, some of which were meaning-altering. The verification of each claim was conducted in a separate annotation process by annotators who were aware of the page but not the sentence from which original claim was extracted and thus in 31.75% of the claims more than one sentence was considered appropriate evidence. Claims require composition of evidence from multiple sentences in 16.82% of cases. Furthermore, in 12.15% of the claims, this evidence was taken from multiple pages. Source: FEVER: a large-scale dataset for Fact Extraction and VERification
MTEB is a benchmark that spans 8 embedding tasks covering a total of 56 datasets and 112 languages. The 8 task types are Bitext mining, Classification, Clustering, Pair Classification, Reranking, Retrieval, Semantic Textual Similarity and Summarisation. The 56 datasets contain varying text lengths and they are grouped into three categories: Sentence to sentence, Paragraph to paragraph, and Sentence to paragraph. Check the latest leaderboards at HuggingFace.
TREC-COVID is a community evaluation designed to build a test collection that captures the information needs of biomedical researchers using the scientific literature during a pandemic. One of the key characteristics of pandemic search is the accelerated rate of change: the topics of interest evolve as the pandemic progresses and the scientific literature in the area explodes. The COVID-19 pandemic provides an opportunity to capture this progression as it happens. TREC-COVID, in creating a test collection around COVID-19 literature, is building infrastructure to support new research and technologies in pandemic search.
The Remote Sensing Image Captioning Dataset (RSICD) is a dataset for remote sensing image captioning task. It contains more than ten thousands remote sensing images which are collected from Google Earth, Baidu Map, MapABC and Tianditu. The images are fixed to 224X224 pixels with various resolutions. The total number of remote sensing images is 10921, with five sentences descriptions per image. Source: https://github.com/201528014227051/RSICD_optimal Image Source: https://github.com/201528014227051/RSICD_optimal
Quora Question Pairs (QQP) dataset consists of over 400,000 question pairs, and each question pair is annotated with a binary value indicating whether the two questions are paraphrase of each other. Source: Bilateral Multi-Perspective Matching for Natural Language Sentences
A new publicly available dataset for verification of climate change-related claims. Source: CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims
The IMAGE-CHAT dataset is a large collection of (image, style trait for speaker A, style trait for speaker B, dialogue between A & B) tuples that we collected using crowd-workers, Each dialogue consists of consecutive turns by speaker A and B. No particular constraints are placed on the kinds of utterance, only that we ask the speakers to both use the provided style trait, and to respond to the given image and dialogue history in an engaging way. The goal is not just to build a diagnostic dataset but a basis for training models that humans actually want to engage with.
TripClick is a large-scale dataset of click logs in the health domain, obtained from user interactions of the Trip Database health web search engine. Provide: * Approximately 5.2 million user interactions * IR evaluation benchmark * Trainin data for deep learning IR models
It is collection regrouping all datasets constituting the ViDoRe benchmark. It includes the test sets from different academic datasets (ArXiVQA, DocVQA, InfoVQA, TATDQA, TabFQuAD) and from datasets synthetically generated spanning various themes and industrial applications: (Artificial Intelligence, Government Reports, Healthcare Industry, Energy and Shift Project). Further details can be found on the corresponding dataset cards.
We present PeerQA, a real-world, scientific, document-level Question Answering (QA) dataset. PeerQA questions have been sourced from peer reviews, which contain questions that reviewers raised while thoroughly examining the scientific article. Answers have been annotated by the original authors of each paper. The dataset contains 579 QA pairs from 208 academic articles, with a majority from ML and NLP, as well as a subset of other scientific communities like Geoscience and Public Health. PeerQA supports three critical tasks for developing practical QA systems: Evidence retrieval, unanswerable question classification, and answer generation. We provide a detailed analysis of the collected dataset and conduct experiments establishing baseline systems for all three tasks. Our experiments and analyses reveal the need for decontextualization in document-level retrieval, where we find that even simple decontextualization approaches consistently improve retrieval performance across architectures. On answer generation, PeerQA serves as a challenging benchmark for long-context modeling, as the papers have an average size of 12k tokens.
A large-scale curated dataset of over 152 million tweets, growing daily, related to COVID-19 chatter generated from January 1st to April 4th at the time of writing. Source: A large-scale COVID-19 Twitter chatter dataset for open scientific research -- an international collaboration
The Belgian Statutory Article Retrieval Dataset (BSARD) is a French native corpus for studying statutory article retrieval. BSARD consists of more than 22,600 statutory articles from Belgian law and about 1,100 legal questions posed by Belgian citizens and labeled by experienced jurists with relevant articles from the corpus.
NFCorpus is a full-text English retrieval data set for Medical Information Retrieval. It contains a total of 3,244 natural language queries (written in non-technical English, harvested from the NutritionFacts.org site) with 169,756 automatically extracted relevance judgments for 9,964 medical documents (written in a complex terminology-heavy language), mostly from PubMed.
LLeQA is a French native dataset for studying information retrieval and long-form question answering in the legal domain. It consists of a knowledge corpus of 27,941 statutory articles collected from the Belgian legislation, and 1,868 legal questions posed by Belgian citizens and labeled by experienced jurists with a comprehensive answer rooted in relevant articles from the corpus.
A novel dataset of 19th-century Latin American press texts, which addresses the lack of specialized corpora for historical and linguistic analysis in this region.
MAPS-KB is a million-scale probabilistic simile knowledge base, covering 4.3 million triplets over 0.4 million terms from 70 GB corpora. It is designed for the tasks of simile detection and component extraction. Source:
Dataset Card for the ACR Appropriateness Criteria Corpus This dataset contains chunked guidelines and narratives from the ACR Appropriateness Criteria, an set of societal guidelines from the American College of Radiology (ACR) to help clinicians order appropriate diagnostic imaging studies for patients. The corpus is formatted similarly to the corpuses introduced in MedRAG by Xiong et al. (2024), and can therefore be similarly used for medical Retrieval-Augmented Generation (RAG). Please abide by the ACR Terms and Conditions when using this dataset. ## Dataset Structure Each row of the dataset contains the following fields: { "id": "A unique identifier for each chunk in the corpus", "title": "The title of the main document from which the chunk was extracted", "content": "The content of the chunk", "contents": "The title and content concatenated together into a single field", "ACRID": "The ACR topic identifier" } Of note, the contents field for each row does not exceed a maximum of 2048 characters in our implementation. ## Dataset Curation This dataset was curated by scraping the ACR Appropriateness Criteria website, downloading all the relevant PDF narratives, and using unstructured's open-source tooling to extract the content into chunks. The original source code used for the dataset curation can be found here. This dataset was constructed on July 17, 2024 - no updates to the ACR Appropriateness Criteria after this date are reflected in the most recent iteration. ## Personal and Sensitive Information The contents of this dataset consist of evidence-based guidelines written by a panel of experts for a medical audience. To our knowledge, no personal or sensitive information is contained in this dataset. ## Dataset Card Contact Please contact Michael Yao at michael.yao@pennmedicine.upenn.edu with any questions or comments.
An evaluation test bed for assessing the robustness of sentence embedding models against user-informed misinformation edits. - A dataset containing perturbed and unperturbed claim pairs used for improving embedding model robustness through knowledge distillation.
CompMix-IR Dataset Overview: Characteristics: CompMix-IR is a heterogeneous knowledge retrieval benchmark dataset, featuring four knowledge types (text, knowledge graphs, tables, and infoboxes), 9,400+ QA pairs, and a corpus of 10 million entries. It supports two retrieval scenarios: retrieving across all knowledge types or retrieving specific types based on user instructions. Motivation: It addresses the limitations of existing benchmarks by providing a more comprehensive and realistic dataset that reflects real-world retrieval needs with diverse knowledge sources and user intents. Potential Use Cases: Ideal for developing and evaluating heterogeneous IR models, instruction-aware retrieval systems, and open-domain QA systems. It can also be used for benchmarking, cross-domain IR research, and enhancing the adaptability and robustness of retrieval models.
CURE is a retrieval dataset with a monolingual and two cross-lingual conditions, with splits spanning ten medical domains. Queries in CURE are natural language questions formulated by healthcare providers. They express the information needs of practitioners consulting academic literature in the course of their duties. Queries are available in English, French and Spanish. The corpus is constructed by mining an index of english passages extracted from biomedical academic articles.
The Multi-Eup is a new multilingual benchmark dataset, comprising 22K multilingual documents collected from the European Parliament, spanning 24 languages. This dataset is designed to investigate fairness in a multilingual information retrieval (IR) context to analyze both language and demographic bias in a ranking context. It boasts an authentic multilingual corpus, featuring topics translated into all 24 languages, as well as cross-lingual relevance judgments. Furthermore, it offers rich demographic information associated with its documents, facilitating the study of demographic bias.
PolyNews is a multilingual parallel dataset containing news titles 833 language pairs, spanning in 64 languages and 17 scripts. PolyNewsParallel aims to provide an easily-accessible, unified and de-duplicated dataset that combines three disparate data sources. It can be used for machine translation or text retrieval in both high-resource and low-resource languages.
Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset