Information Retrieval

The Natural Questions corpus is a question answering dataset containing 307,373 training examples, 7,830 development examples, and 7,842 test examples. Each example is comprised of a google.com query and a corresponding Wikipedia page. Each Wikipedia page has a passage (or long answer) annotated on the page that answers the question and one or more short spans from the annotated passage containing the actual answer. The long and the short answer annotations can however be empty. If they are both empty, then there is no answer on the page at all. If the long answer annotation is non-empty, but the short answer annotation is empty, then the annotated passage answers the question but no explicit short answer could be found. Finally 1% of the documents have a passage annotated with a short answer that is “yes” or “no”, instead of a list of short spans. Source: A BERT Baseline for the Natural Questions Image Source: https://paperswithcode.com/paper/natural-questions-a-benchmark-for-question/

The MS MARCO (Microsoft MAchine Reading Comprehension) is a collection of datasets focused on deep learning in search. The first dataset was a question answering dataset featuring 100,000 real Bing questions and a human generated answer. Over time the collection was extended with a 1,000,000 question dataset, a natural language generation dataset, a passage ranking dataset, keyphrase extraction dataset, crawling dataset, and a conversational search. Source: https://microsoft.github.io/msmarco/ Image Source: https://arxiv.org/pdf/1809.08267.pdf

CoNLL-2003 is a named entity recognition dataset released as a part of CoNLL-2003 shared task: language-independent named entity recognition. The data consists of eight files covering two languages: English and German. For each of the languages there is a training file, a development file, a test file and a large file with unannotated data. The English data was taken from the Reuters Corpus. This corpus consists of Reuters news stories between August 1996 and August 1997. For the training and development set, ten days worth of data were taken from the files representing the end of August 1996. For the test set, the texts were from December 1996. The preprocessed raw data covers the month of September 1996. The text for the German data was taken from the ECI Multilingual Text Corpus. This corpus consists of texts in many languages. The portion of data that was used for this task, was extracted from the German newspaper Frankfurter Rundshau. All three of the training, development and test sets were taken from articles written in one week at the end of August 1992. The raw data were taken from the months of September to December 1992. | English data | Articles | Sentences | Tokens | LOC | MISC | ORG | PER | |-------------------|----------|-----------|---------|------|------|------|------| | Training set | 946 | 14,987 | 203,621 | 7140 | 3438 | 6321 | 6600 | | Development set | 216 | 3,466 | 51,362 | 1837 | 922 | 1341 | 1842 | | Test set | 231 | 3,684 | 46,435 | 1668 | 702 | 1661 | 1617 | Number of articles, sentences, tokens and entities (locations, miscellaneous, organizations, and persons) in English data files. | German data | Articles | Sentences | Tokens | LOC | MISC | ORG | PER | |-------------------|----------|-----------|---------|------|------|------|------| | Training set | 553 | 12,705 | 206,931 | 4363 | 2288 | 2427 | 2773 | | Development set | 201 | 3,068 | 51,444 | 1181 | 1010 | 1241 | 1401 | | Test set | 155 | 3,160 | 51,943 | 1035 | 670 | 773 | 1195 | Number of articles, sentences, tokens and entities (locations, miscellaneous, organizations, and persons) in German data files.

BEIR (Benchmarking IR) is a heterogeneous benchmark containing different information retrieval (IR) tasks. Through BEIR, it is possible to systematically study the zero-shot generalization capabilities of multiple neural retrieval approaches. The benchmark contains a total of 9 information retrieval tasks (Fact Checking, Citation Prediction, Duplicate Question Retrieval, Argument Retrieval, News Retrieval, Question Answering, Tweet Retrieval, Biomedical IR, Entity Retrieval) from 19 different datasets: * MS MARCO * TREC-COVID * NFCorpus * BioASQ * Natural Questions * HotpotQA * FiQA-2018 * Signal-1M * TREC-News * ArguAna * Touche 2020 * CQADupStack * Quora Question Pairs * DBPedia * SciDocs * FEVER * Climate-FEVER * SciFact * Robust04

BioASQ is a question answering dataset. Instances in the BioASQ dataset are composed of a question (Q), human-annotated answers (A), and the relevant contexts (C) (also called snippets). Source: Transferability of Natural Language Inference to Biomedical Question Answering Image Source: http://participants-area.bioasq.org/datasets/

CORD-19 is a free resource of tens of thousands of scholarly articles about COVID-19, SARS-CoV-2, and related coronaviruses for use by the global research community. Source: Allen Institute for AI

MTEB is a benchmark that spans 8 embedding tasks covering a total of 56 datasets and 112 languages. The 8 task types are Bitext mining, Classification, Clustering, Pair Classification, Reranking, Retrieval, Semantic Textual Similarity and Summarisation. The 56 datasets contain varying text lengths and they are grouped into three categories: Sentence to sentence, Paragraph to paragraph, and Sentence to paragraph. Check the latest leaderboards at HuggingFace.

The Free Music Archive (FMA) is a large-scale dataset for evaluating several tasks in Music Information Retrieval. It consists of 343 days of audio from 106,574 tracks from 16,341 artists and 14,854 albums, arranged in a hierarchical taxonomy of 161 genres. It provides full-length and high-quality audio, pre-computed features, together with track- and user-level metadata, tags, and free-form text such as biographies. There are four subsets defined by the authors: * Full: the complete dataset, * Large: the full dataset with audio limited to 30 seconds clips extracted from the middle of the tracks (or entire track if shorter than 30 seconds), * Medium: a selection of 25,000 30s clips having a single root genre, * Small: a balanced subset containing 8,000 30s clips with 1,000 clips per one of 8 root genres. The official split into training, validation and test sets (80/10/10) uses stratified sampling to preserve the percentage of tracks per genre. Songs of the same artists are part of one set only. Source: FMA: A Dataset For Music Analysis Audio Source: https://github.com/mdeff/fma

PubLayNet is a dataset for document layout analysis by automatically matching the XML representations and the content of over 1 million PDF articles that are publicly available on PubMed Central. The size of the dataset is comparable to established computer vision datasets, containing over 360 thousand document images, where typical document layout elements are annotated. Source: PubLayNet: largest dataset ever for document layout analysis

QASC is a question-answering dataset with a focus on sentence composition. It consists of 9,980 8-way multiple-choice questions about grade school science (8,134 train, 926 dev, 920 test), and comes with a corpus of 17M sentences. Source: Allen Institute for AI

TREC-COVID is a community evaluation designed to build a test collection that captures the information needs of biomedical researchers using the scientific literature during a pandemic. One of the key characteristics of pandemic search is the accelerated rate of change: the topics of interest evolve as the pandemic progresses and the scientific literature in the area explodes. The COVID-19 pandemic provides an opportunity to capture this progression as it happens. TREC-COVID, in creating a test collection around COVID-19 literature, is building infrastructure to support new research and technologies in pandemic search.

MedleyDB, is a dataset of annotated, royalty-free multitrack recordings. It was curated primarily to support research on melody extraction. For each song melody f₀ annotations are provided as well as instrument activations for evaluating automatic instrument recognition. The original dataset consists of 122 multitrack songs out of which 108 include melody annotations. The songs in MedleyDB were obtained from the following sources: * Independent Artists (30 songs) * NYU's Dolan Recording Studio (32 songs) * Weathervane Music (25 songs) * Music Delta (35 songs) MedleyDB contains songs of a variety of musical genres: Singer/Songwriter, Classical, Rock, World/Folk, Fusion, Jazz, Pop, Musical Theatre, Rap. For each song three types of audio content are given: a mix, stems, and raw audio. All types of audio files are .wav files with a sample rate of 44.1 kHz and a bit depth of 16. Source: https://medleydb.weebly.com/ Image Source: https://medleydb.weebly.com/ Audio Source: https://zenodo.org/record/1438309

A large-scale MultiLingual SUMmarization dataset. Obtained from online newspapers, it contains 1.5M+ article/summary pairs in five different languages -- namely, French, German, Spanish, Russian, Turkish. Together with English newspapers from the popular CNN/Daily mail dataset, the collected data form a large scale multilingual dataset which can enable new research directions for the text summarization community. Source: MLSUM: The Multilingual Summarization Corpus

InsuranceQA is a question answering dataset for the insurance domain, the data stemming from the website Insurance Library. There are 12,889 questions and 21,325 answers in the training set. There are 2,000 questions and 3,354 answers in the validation set. There are 2,000 questions and 3,308 answers in the test set. Source: APPLYING DEEP LEARNING TO ANSWER SELECTION: A STUDY AND AN OPEN TASK

The MSLR-WEB10K dataset consists of 10,000 search queries over the documents from search results. The data also contains the values of 136 features and a corresponding user-labeled relevance factor on a scale of one to five with respect to each query-document pair. It is a subset of the MSLR-WEB30K dataset. Source: Dueling Bandits with Qualitative Feedback

SciTSR is a large-scale table structure recognition dataset, which contains 15,000 tables in PDF format and their corresponding structure labels obtained from LaTeX source files. Source: https://github.com/Academic-Hammer/SciTSR

The MQ2007 dataset consists of queries, corresponding retrieved documents and labels provided by human experts. The possible relevance labels for each document are “relevant”, “partially relevant”, and “not relevant”. Source: ARSM GRADIENT ESTIMATOR FOR SUPERVISED LEARNING TO RANK

GuitarSet is a dataset of high-quality guitar recordings and rich annotations. It contains 360 excerpts 30 seconds in length. The 360 excerpts are the result of the following combinations: * 6 players, * 2 versions: comping and soloing, * 5 styles: Rock, Singer-Songwriter, Bossa Nova, Jazz, and Funk, * 3 progressions: 12 Bar Blues, Autumn Leaves, and Pachelbel Canon, * 2 tempi: slow and fast. Each excerpt is annotated with 6 pitch contour and midi note annotations (one per string), 2 chord annotations (instructed and performed), beat and tempo annotations. Source: https://guitarset.weebly.com/ Audio Source: https://zenodo.org/record/3371780

The MQ2008 dataset is a dataset for Learning to Rank. It contains 800 queries with labelled documents.

The MSLR-WEB30K dataset consists of 30,000 search queries over the documents from search results. The data also contains the values of 136 features and a corresponding user-labeled relevance factor on a scale of one to five with respect to each query-document pair.

EntityQuestions is a dataset of simple, entity-rich questions based on facts from Wikidata (e.g., "Where was Arve Furset born? ").

WikiReading is a large-scale natural language understanding task and publicly-available dataset with 18 million instances. The task is to predict textual values from the structured knowledge base Wikidata by reading the text of the corresponding Wikipedia articles. The task contains a rich variety of challenging classification and extraction sub-tasks, making it well-suited for end-to-end models such as deep neural networks (DNNs). Source: WIKIREADING: A Novel Large-scale Language Understanding Task over Wikipedia Image Source: Hewlett et al

A large scale dataset to enable the transfer step, exploiting the Natural Questions dataset. Source: TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection

The Yahoo! Learning to Rank Challenge dataset consists of 709,877 documents encoded in 700 features and sampled from query logs of the Yahoo! search engine, spanning 29,921 queries. Source: Ranking for Relevance and Display Preferencesin Complex Presentation Layouts

Composed of 1,395 questions posed by crowdworkers on Wikipedia articles, and a machine translation of the Stanford Question Answering Dataset (Arabic-SQuAD). Source: Neural Arabic Question Answering

HeadQA is a multi-choice question answering testbed to encourage research on complex reasoning. The questions come from exams to access a specialized position in the Spanish healthcare system, and are challenging even for highly specialized humans. Source: HEAD-QA: A Healthcare Dataset for Complex Reasoning Image Source: https://arxiv.org/pdf/1906.04701v1.pdf

The George Washington dataset contains 20 pages of letters written by George Washington and his associates in 1755 and thereby categorized into historical collection. The images are annotated at word level and contain approximately 5,000 words. Source: HWNet v2: An Efficient Word Image Representation for Handwritten Documents. Image Source: https://www.loc.gov/resource/mgw1a.002/?sp=2

The iKala dataset is a singing voice separation dataset that comprises of 252 30-second excerpts sampled from 206 iKala songs (plus 100 hidden excerpts reserved for MIREX data mining contest). The music accompaniment and the singing voice are recorded at the left and right channels respectively. Additionally, the human-labeled pitch contours and timestamped lyrics are provided. This dataset is not available anymore. Source: http://mac.citi.sinica.edu.tw/ikala/ 1. "T" as a sofa: The "T" horizontal strip can mimic the back of a sofa with a delicate cushion or details of the uphols or appliances with the color button. The "T" vertical strip can show a feet or arm of the sofa, shiny, yet firm. 2. Merge "P": Put "P" next to "T", your curve to delicately with the top "T." It is intertwined. The circular part of "P" can show a cushion or a curved chair and synchronize the subject of furniture. Make sure "P" is visually relying on "T", which reflects the relationship of cohesion and balance. 5. Coherence of "B" and "I": "B" can be aligned as a pair of cushions or a modern chair, with mild curves with glossy and modern aesthetics. "I" can be a symbol of a shiny furniture or a vertical light bar and completes the shapes without overburdess them. Color palette 4: Includes soft soil colors such as beige, top and gray shades, along with silent or silver gold tips to touch elegance. Consider a slope effect to enhance modernity, to keep colors elegant and complex. 5. Connect the letters: Use the overlap or intertwined edges that the letters meet for the symbol of unity. The plan should allow viewers to distinguish each letter while feeling part of the same "structure". 6. Background patterns: Use delicate geometric patterns or textures that mimic fabrics or furniture materials such as wood seeds or woven fibers. These patterns must remain minimalist and focus on highlighting the logo, while maintaining communication. While it deals with the subject of furniture and design, this concept conveys modernity, creativity and professional. If you like, I can create a draft design for better visualization.

QAMPARI is an ODQA benchmark, where question answers are lists of entities, spread across many paragraphs. It was created by (a) generating questions with multiple answers from Wikipedia's knowledge graph and tables, (b) automatically pairing answers with supporting evidence in Wikipedia paragraphs, and (c) manually paraphrasing questions and validating each answer. Source: QAMPARI: : An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs Image Source: https://arxiv.org/pdf/2205.12665v2.pdf

A dataset on asking Questions for Lack of Clarity in open-domain information-seeking conversations. Qulac presents the first dataset and offline evaluation framework for studying clarifying questions in open-domain information-seeking conversational search systems. Source: https://github.com/aliannejadi/qulac Image Source: https://github.com/aliannejadi/qulac

A dataset with 2,437 dialogues and 10,917 QA pairs. The dialogues are collected from three Stack Exchange sites using the Wizard of Oz method with crowdsourcing. Source: DoQA -- Accessing Domain-Specific FAQs via Conversational QA

ORCAS is a click-based dataset. It covers 1.4 million of the TREC DL documents, providing 18 million connections to 10 million distinct queries.

Enhances QuAC by adapting it to an open-retrieval setting. It is an aggregation of three existing datasets: (1) the QuAC dataset that offers information-seeking conversations, (2) the CANARD dataset that consists of context-independent rewrites of QuAC questions, and (3) the Wikipedia corpus that serves as the knowledge source of answering questions. Source: ORConvQA

JuICe is a corpus of 1.5 million examples with a curated test set of 3.7K instances based on online programming assignments. Compared with existing contextual code generation datasets, JuICe provides refined human-curated data, open-domain code, and an order of magnitude more training data. Source: JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation

TripClick is a large-scale dataset of click logs in the health domain, obtained from user interactions of the Trip Database health web search engine. Provide: * Approximately 5.2 million user interactions * IR evaluation benchmark * Trainin data for deep learning IR models

The beginnings of a question answering dataset specifically designed for COVID-19, built by hand from knowledge gathered from Kaggle's COVID-19 Open Research Dataset Challenge. Source: Rapidly Bootstrapping a Question Answering Dataset for COVID-19