Papers with Code 아카이브

Information Retrieval

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
6
평가표
33
보존된 평가 기록
17

Information retrieval is the task of ranking a list of documents or search results in response to a query

<span style="color:grey; opacity: 0.6">( Image credit: sudhanshumittal )</span>

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

ResQ

ReSQ is a real-world Spatial Question Answering dataset with human-generated questions built on an existing corpus with SpRL annotations. This dataset can be used to evaluate spatial language processing models in realistic situations.

RP2K

A new large-scale retail product dataset for fine-grained image classification. Unlike previous datasets focusing on relatively few products, more than 500,000 images of retail products on shelves were collected, belonging to 2000 different products. The dataset aims to advance the research in retail object recognition, which has massive applications such as automatic shelf auditing and image-based product information retrieval. Source: RP2K: A Large-Scale Retail Product Dataset for Fine-Grained Image Classification

TREC-News

The TREC News Track features modern search tasks in the news domain. In partnership with The Washington Post, we are developing test collections that support the search needs of news readers and news writers in the current news environment. It's our hope that the track will foster research that establishes a new sense for what "relevance" means for news search.

TriviaHG

Nowadays, individuals tend to engage in dialogues with Large Language Models, seeking answers to their questions. In times when such answers are readily accessible to anyone, the stimulation and preservation of human’s cognitive abilities, as well as the assurance of maintaining good reasoning skills by humans becomes crucial. This study addresses such needs by proposing hints (instead of final answers or before giving answers) as a viable solution. We introduce a framework for the automatic hint generation for factoid questions, employing it to construct TriviaHG, a novel large-scale dataset featuring 160,230 hints corresponding to 16,645 questions from the TriviaQA dataset. Additionally, we present an automatic evaluation method that measures the Convergence and Familiarity quality attributes of hints. To evaluate the TriviaHG dataset and the proposed evaluation method, we enlisted 10 individuals to annotate 2,791 hints and tasked 6 humans with answering questions using the provided hints. The effectiveness of hints varied, with success rates of 96%, 78%, and 36% for questions with easy, medium, and hard answers, respectively. Moreover, the proposed automatic evaluation methods showed a robust correlation with annotators’ results. Conclusively, the findings highlight three key insights: the facilitative role of hints in resolving unknown questions, the dependence of hint quality on answer difficulty, and the feasibility of employing automatic evaluation methods for hint assessment.

ArxivPapers

The ArxivPapers dataset is an unlabelled collection of over 104K papers related to machine learning and published on arXiv.org between 2007–2020. The dataset includes around 94K papers (for which LaTeX source code is available) in a structured form in which paper is split into a title, abstract, sections, paragraphs and references. Additionally, the dataset contains over 277K tables extracted from the LaTeX papers. Due to the papers license the dataset is published as a metadata and open-source pipeline that can be used to obtain and convert the papers. Source: AxCell: Automatic Extraction of Results from Machine Learning Papers Image Source: AxCell: Automatic Extraction of Results from Machine Learning Papers

BoostCLIR

BoostCLIR is a bilingual (Japanese-English) corpus of patent abstracts, extracted from the MAREC patent data, and the data from the NTCIR PatentMT workshop collections, accompanied with relevance judgements for the task of patent prior-art search. Important: The English side of the corpus contains patent IDs as well as the text of the abstracts. The Japanese side only contains patent IDs because of NTCIR copyright restrictions. The Japanese patent abstracts can be extracted from full text Japanese patent documents, which are available from the organizers of the NTCIR workshop.

GermanDPR

GermanDPR is a dataset for passage retrieval in German. GermanDPR comprises 8,245 question/answer pairs in the training set, 1,030 pairs in the development set, and 1,025 pairs in the test set. For each pair, there are one positive context and three hard negative contexts.

Large-Scale CLIR Dataset

The Large-Scale CLIR Dataset is a retrieval dataset built for Cross-Language Information Retrieval (CLIR). The dataset is derived from Wikipedia and contains more 2.8 million English single-sentence queries with relevant documents from 25 other selected languages.

ORCAS-I

A labelled version of the ORCAS click-based dataset of Web queries, which provides 18 million connections to 10 million distinct queries. DOI of the dataset: 10.48436/pp7xz-n9a06

SegmentedTables

The SegmentedTables dataset is a collection of almost 2,000 tables extracted from 352 machine learning papers. Each table consists of rich text content, layout and caption. Tables are annotated with types (leaderboard, ablation, irrelevant) and cells of relevant tables are annotated with semantic roles (such as “paper model”, “competing model”, “dataset”, “metric”). Due to the license of source papers the dataset is published as a metadata, all annotations and open-source pipeline that can be used to extract the tables. Source: AxCell: Automatic Extraction of Results from Machine Learning Papers Image Source: AxCell: Automatic Extraction of Results from Machine Learning Papers

ThreatGram 101 - Extreme Telegram Data

Data 1: Raw and Unlabeled; 2 million unlabeled replies from 17 Telegram channels. Data 2: Raw and Labeled; 15,076 replies from 17 Telegram channels categorized as no threat, judicial threat, and non-judicial threat.

Twitter Conversations Dataset

This dataset is used for the task of conversational document prediction. The dataset includes conversations that occurred between users and customer care agents in 25 organizations on the Twitter platform. Each conversation ends with a customer care agent providing a URL to a document to resolve the issue the user is facing. The task is to predict the document given a dialog context. The train, dev and test datasets include 10000, 525 and 500 conversations respectively. Source: https://github.com/IBM/twitter-customer-care-document-prediction Image Source: https://arxiv.org/pdf/2010.02305v1.pdf

Webis-Touché-2020

This paper is a condensed report on the second year of the Touché shared task on argument retrieval held at CLEF 2021. With the goal to provide a collaborative platform for researchers, we organized two tasks: (1) supporting individuals in finding arguments on controversial topics of social importance and (2) supporting individuals with arguments in personal everyday comparison situations.

CompMix-IR

CompMix-IR Dataset Overview: Characteristics: CompMix-IR is a heterogeneous knowledge retrieval benchmark dataset, featuring four knowledge types (text, knowledge graphs, tables, and infoboxes), 9,400+ QA pairs, and a corpus of 10 million entries. It supports two retrieval scenarios: retrieving across all knowledge types or retrieving specific types based on user instructions. Motivation: It addresses the limitations of existing benchmarks by providing a more comprehensive and realistic dataset that reflects real-world retrieval needs with diverse knowledge sources and user intents. Potential Use Cases: Ideal for developing and evaluating heterogeneous IR models, instruction-aware retrieval systems, and open-domain QA systems. It can also be used for benchmarking, cross-domain IR research, and enhancing the adaptability and robustness of retrieval models.

CoSQA+

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

CSPRD

The Chinese Stock Policy Retrieval Dataset (CSPRD) contains a Chinese policy corpus of 10,002 articles and 709 prospectus examples from 545 companies listed on China’s Science and Technology Innovation Board (STAR Market). CSPRD is bilingual in Chinese and English (Translated by ChatGPT) and is annotated by experienced experts from Shanghai Stock Exchange.

DAPFAM

Dataset DAPFAM See the accompanying paper: Ayaou et al., 2025 — “DAPFAM: A Domain‑Aware Patent Retrieval Dataset Aggregated at the Family Level” (arXiv:2506.22141). ## Summary DAPFAM provides 1 247 balanced query patent families and 45 336 target families with forward/backward‑citation relevance labels (≈ 50 K pairs). Each relevant link is explicitly marked in‑domain or out‑of‑domain according to IPC 3‑char overlap, enabling rigorous cross‑domain evaluation. * Full text (title · abstract · claims · description) plus rich metadata for every family. * Multi‑jurisdictional, English‑only text (families may originate in US, JP, EP, CN, …). * Parquet qrel file: qrels_all.parquet. ## Dataset Structure corpus.parquet # 45 336 rows, targets – every original column from the paper queries.parquet # 1 247 rows, queries – same columns + abstract_keywords qrels_all.parquet # (all | in | out) four‑column tables → query_id · relevant_id · relevance_score · domain_rel ## Citation @misc{ayaou2025dapfam, title={DAPFAM: A Domain-Aware Patent Retrieval Dataset Aggregated at the Family Level}, author={Iliass Ayaou and Denis Cavallucci and Hicham Chibane}, year={2025}, eprint={2506.22141}, archivePrefix={arXiv}, primaryClass={cs.CL} } ## Quick Stats * Queries: 1,247 * Corpus (targets): 45,336 * Qrels (all): 49,869 * Qrels (in): 19,736 * Qrels (out): 5,193

diaforge-utc-r-0725

Dataset for our paper Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky which includes 5000 enterprise tools and the corresponding dialogues generated using DiaFORGE UTC data engine. The dataset is generated with the data generation engine. The engine simulates a user agent and an assistant agent in a dialogue, where the user agent has a persona and the assistant agent has access to a set of tools. Detailed information about the data generation process can be found in the paper. Each entry in the dataset contains: - seed: Gold tool for the generated dialogue. - user_persona: Persona of the simulated user agent. - messages: List of messages in the dialogue, where each message is a dictionary containing: - distractor_tools: List of distractor tools that are not the gold tool but are relevant to the dialogue. These tools are used by user agent to generate utterances that are hard to disambiguate from the gold tool. - retrieved_tools: List of tools retrieved by the assistant agent and used by the assistant agent in order to ask clarifying questions to the user agent.

FZ queries

A set of 248 search queries annotated with the correct diagnosis. The diagnosis is referenced with a Concept Unique Identifier (CUI). In a retrieval setting, the task consists of retrieving an article from the FindZebra corpus with a CUI that matches the query CUI.

GLARE

The Guided Lexrank algorithm is applied to dataset special_appeal.csv to summarize the texts of legal documents. The obtained summary and the texts of the topics contained in dataset themes.csv are submitted to the BM25 algorithm for similarity assessment. From a list of topics, the GLARE method produces a ranking with suggested topics for a given document.

LinkedResults

The LinkedResults dataset contains around 1,600 results capturing performance of machine learning models from tables of 239 papers. All tables come from a subset of SegmentedTables dataset. Each result is a tuple of form (task, dataset, metric name, metric value) and is linked to a particular table, row and cell it originates from. Source: AxCell: Automatic Extraction of Results from Machine Learning Papers

MetaCLIR

This data adds textual meta-infomation data to two existing corpora for cross language information retrieval: BoostCLIR, and the Large Scale CLIR Dataset (wiki-clir).

OpenDebateEvidence

We introduce OpenDebateEvidence, a comprehensive dataset for argument mining and summarization sourced from the American Competitive Debate community. This dataset includes over 3.5 million documents with rich metadata, making it one of the most extensive collections of debate evidence. OpenDebateEvidence captures the complexity of arguments in high school and college debates, pro- viding valuable resources for training and evaluation. Our extensive experiments demonstrate the efficacy of fine-tuning state-of-the-art large language models for argumentative abstractive summarization across various methods, models, and datasets. By providing this comprehensive resource, we aim to advance com- putational argumentation and support practical applications for debaters, edu- cators, and researchers. OpenDebateEvidence is publicly available to support further research and innovation in computational argumentation. Access it here: https://huggingface.co/datasets/Yusuf5/OpenCaselist

Persian Reverse Dictionary Dataset

The Persian Reverse Dictionary Dataset is a collection of 855217 words along with the phrases describing them. The phrases were extracted from the top three most well-known Persian dictionaries (including Amid, Moeen, and Dehkhoda), Persian Wikipedia, and a Persian Wordnet (called Farsnet).

Phrase-in-Context

Phrase in Context is a curated benchmark for phrase understanding and semantic search, consisting of three tasks of increasing difficulty: Phrase Similarity (PS), Phrase Retrieval (PR) and Phrase Sense Disambiguation (PSD). The datasets are annotated by 13 linguistic experts on Upwork and verified by two groups: ~1000 AMT crowdworkers and another set of 5 linguistic experts. PiC benchmark is distributed under CC-BY-NC 4.0.

Pylon Benchmark

We create a new dataset from GitTables, a data lake of 1.7M tables extracted from CSV files on GitHub. The benchmark comprises 1,746 tables including union-able table subsets under topics selected from Schema.org: scholarly article, job posting, and music playlist. We end up with these three topics since we can find a fair number of union-able tables of them from diverse sources in the corpus (we can easily find union-able tables from a single source but they are less interesting for table union search as simple syntactic methods can identify all of them because of the same schema and consistent value representations).

Urdu MsMarco

This dataset is the translation of the MS-marco dataset, marking it the first large-scale urdu IR dataset. Dataset Details: The MS MARCO dataset is formed by a collection of 8.8M passages, approximately 530k queries, and at least one relevant passage per query, which were selected by humans. The development set of MS MARCO comprises more than 100k queries. However, a smaller set of 6,980 queries is used for evaluation in most published works. The triples files (triples.train.small.urdu.tsv , named as triple files part aa to ae) is around 47 GB and is split into 5 parts. Download them and combine them before you start working. This dataset is created using the IndicTrans2 translation model.

Urdu News Headlines Dataset

Urdu News Headlines Dataset with VOA and BBC An Urdu news headlines dataset is a collection of news headlines in the Urdu language, typically scraped from news websites and social media platforms. These datasets can be valuable for researchers and developers working on a variety of tasks, such as: Machine translation: Training machine translation models to translate between Urdu and other languages. Text summarization: Developing algorithms for automatically summarizing Urdu news articles. Natural language processing: Studying the structure and grammar of the Urdu language. Clustering: Grouping news articles into similar categories based on their headlines. Benefits of using a dataset with VOA and BBC headlines There are several benefits to using a dataset that includes news headlines from VOA and BBC: High quality: VOA and BBC are well-respected news organizations known for their high-quality journalism. This means that the headlines in the dataset are likely to be accurate and unbiased. Diversity of topics: VOA and BBC cover a wide range of topics, including politics, business, sports, and entertainment. This means that the dataset will be representative of the different types of news that are available in the Urdu language. Large size: VOA and BBC have been publishing news in Urdu for many years. This means that there is a large amount of data available, which can be used to train machine learning models and to conduct research. Here are some examples of how a Urdu news headlines dataset with VOA and BBC headlines can be used: A machine translation model could be trained on the dataset to translate Urdu news headlines into English. This could be used to make Urdu news more accessible to a wider audience. A text summarization algorithm could be trained on the dataset to automatically generate summaries of Urdu news articles. This could be used to save people time and to help them to stay informed. A clustering algorithm could be used to group Urdu news articles into similar categories based on their headlines. This could be used to create personalized news feeds for users. Here are some limitations of using a Urdu news headlines dataset with VOA and BBC headlines: Limited scope: VOA and BBC are primarily English-language news organizations. This means that the dataset may not be representative of the full range of Urdu news that is available. Bias: VOA and BBC are both Western news organizations. This means that the dataset may be biased towards Western perspectives. Limited access: The data may not be readily available to everyone.

WikiPII

WikiPII, an automatically labeled dataset composed of Wikipedia biography pages, annotated for personal information extraction.

WMT 2014 Medical

The Medical Translation Task of WMT 2014 addresses the problem of domain-specific and genre-specific machine translation. The task is split into two subtasks: summary translation, focused on translation of sentences from summaries of medical articles, and query translation, focused on translation of queries entered by users into medical information search engines. Both subtasks included translation between English and Czech, German, and French, in both directions. Source: https://www.aclweb.org/anthology/W14-3302.pdf

CLEF-TAR

Technologically Assisted Reviews in Empirical Medicine.

NIAN

The Needle in a Needlestack (NIAN) is a new benchmark designed to measure how well Language Learning Models (LLMs) pay attention to the information in their context window¹. In this benchmark, a prompt is created that includes thousands of limericks, and the prompt asks a question about one limerick at a specific location¹. The goal is to test the ability of LLMs to locate and understand specific information within a large context. For example, a prompt might include around 2500 limericks, and the LLM is asked a question about a limerick at a certain position¹. The LLM's task is to correctly answer the question by paying attention to the right part of the context. (1) GPT-4o’s Memory Breakthrough! (NIAN code) | needle-in-a-needlestack. http://nian.llmonpy.ai/. (2) [P] Needle in a Needlestack (NIAN) | allainews.com. https://allainews.com/item/p-needle-in-a-needlestack-nian-2024-05-16/. (3) GitHub - llmonpy/needle-in-a-needlestack. https://github.com/llmonpy/needle-in-a-needlestack/.

PANACEA

The peer-reviewed publication for this dataset has been presented in the 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), and can be accessed here: https://arxiv.org/abs/2205.02596. Please cite this when using the dataset. This dataset contains a heterogeneous set of True and False COVID claims and online sources of information for each claim. The claims have been obtained from online fact-checking sources, existing datasets and research challenges. It combines different data sources with different foci, thus enabling a comprehensive approach that combines different media (Twitter, Facebook, general websites, academia), information domains (health, scholar, media), information types (news, claims) and applications (information retrieval, veracity evaluation). The processing of the claims included an extensive de-duplication process eliminating repeated or very similar claims. The dataset is presented in a LARGE and a SMALL version, accounting for different degrees of similarity between the remaining claims (excluding respectively claims with a 90% and 99% probability of being similar, as obtained through the MonoT5 model). The similarity of claims was analysed using BM25 (Robertson et al., 1995; Crestani et al., 1998; Robertson and Zaragoza, 2009) with MonoT5 re-ranking (Nogueira et al., 2020), and BERTScore (Zhang et al., 2019). The processing of the content also involved removing claims making only a direct reference to existing content in other media (audio, video, photos); automatically obtained content not representing claims; and entries with claims or fact-checking sources in languages other than English. The claims were analysed to identify types of claims that may be of particular interest, either for inclusion or exclusion depending on the type of analysis. The following types were identified: (1) Multimodal; (2) Social media references; (3) Claims including questions; (4) Claims including numerical content; (5) Named entities, including: PERSON − People, including fictional; ORGANIZATION − Companies, agencies, institutions, etc.; GPE − Countries, cities, states; FACILITY − Buildings, highways, etc. These entities have been detected using a RoBERTa base English model (Liu et al., 2019) trained on the OntoNotes Release 5.0 dataset (Weischedel et al., 2013) using Spacy. The original labels for the claims have been reviewed and homogenised from the different criteria used by each original fact-checker into the final True and False labels. The data sources used are: - The CoronaVirusFacts/DatosCoronaVirus Alliance Database. https://www.poynter.org/ifcn-covid-19-misinformation/ - CoAID dataset (Cui and Lee, 2020) https://github.com/cuilimeng/CoAID - MM-COVID (Li et al., 2020) https://github.com/bigheiniu/MM-COVID - CovidLies (Hossain et al., 2020) https://github.com/ucinlp/covid19-data - TREC Health Misinformation track https://trec-health-misinfo.github.io/ - TREC COVID challenge (Voorhees et al., 2021; Roberts et al., 2020) https://ir.nist.gov/covidSubmit/data.html The LARGE dataset contains 5,143 claims (1,810 False and 3,333 True), and the SMALL version 1,709 claims (477 False and 1,232 True). The entries in the dataset contain the following information: - Claim. Text of the claim. - Claim label. The labels are: False, and True. - Claim source. The sources include mostly fact-checking websites, health information websites, health clinics, public institutions sites, and peer-reviewed scientific journals. - Original information source. Information about which general information source was used to obtain the claim. - Claim type. The different types, previously explained, are: Multimodal, Social Media, Questions, Numerical, and Named Entities. Funding. This work was supported by the UK Engineering and Physical Sciences Research Council (grant no. EP/V048597/1, EP/T017112/1). ML and YH are supported by Turing AI Fellowships funded by the UK Research and Innovation (grant no. EP/V030302/1, EP/V020579/1). References - Arana-Catania M., Kochkina E., Zubiaga A., Liakata M., Procter R., He Y.. Natural Language Inference with Self-Attention for Veracity Assessment of Pandemic Claims. NAACL 2022 https://arxiv.org/abs/2205.02596 - Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp,109:109. - Fabio Crestani, Mounia Lalmas, Cornelis J Van Rijsbergen, and Iain Campbell. 1998. “is this document relevant?. . . probably” a survey of probabilistic models in information retrieval. ACM Computing Surveys (CSUR), 30(4):528–552. - Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Now Publishers Inc. - Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre-trained sequence-to-sequence model. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 708–718. - Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. - Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. - Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23. - Limeng Cui and Dongwon Lee. 2020. Coaid: Covid-19 healthcare misinformation dataset. arXiv preprint arXiv:2006.00885. - Yichuan Li, Bohan Jiang, Kai Shu, and Huan Liu. 2020. Mm-covid: A multilingual and multimodal data repository for combating covid-19 disinformation. - Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. COVIDLies: Detecting COVID-19 misinformation on social media. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online. Association for Computational Linguistics. - Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing a pandemic information retrieval test collection. In ACM SIGIR Forum, volume 54, pages 1–12. ACM New York, NY, USA.

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.