Entity Resolution

The Amazon-Google dataset for entity resolution derives from the online retailers Amazon.com and the product search service of Google accessible through the Google Base Data API. The dataset contains 1363 entities from amazon.com and 3226 google products as well as a gold standard (perfect mapping) with 1300 matching record pairs between the two data sources. The common attributes between the two data sources are: product name, product description, manufacturer and price. The dataset was initially published in the repository of the Database Group of the University of Leipzig: https://dbs.uni-leipzig.de/research/projects/object_matching/benchmark_datasets_for_entity_resolution To enable the reproducibility of the results and the comparability of the performance of different matchers on the Amazon-Google matching task, the dataset was split into fixed train, validation and test sets. The fixed splits are provided in the CompERBench repository: http://data.dws.informatik.uni-mannheim.de/benchmarkmatchingtasks/index.html

The Abt-Buy dataset for entity resolution derives from the online retailers Abt.com and Buy.com. The dataset contains 1081 entities from abt.com and 1092 entities from buy.com as well as a gold standard (perfect mapping) with 1097 matching record pairs between the two data sources. The common attributes between the two data sources are: product name, product description and product price. The dataset was initially published in the repository of the Database Group of the University of Leipzig: https://dbs.uni-leipzig.de/research/projects/object_matching/benchmark_datasets_for_entity_resolution To enable the reproducibility of the results and the comparability of the performance of different matchers on the Abt-Buy matching task, the dataset was split into fixed train, validation and test sets. The fixed splits are provided in the CompERBench repository: http://data.dws.informatik.uni-mannheim.de/benchmarkmatchingtasks/index.html

Many e-shops have started to mark-up product data within their HTML pages using the schema.org vocabulary. The Web Data Commons project regularly extracts such data from the Common Crawl, a large public web crawl. The Web Data Commons Training and Test Sets for Large-Scale Product Matching contain product offers from different e-shops in the form of binary product pairs (with corresponding label "match" or "no match") for four product categories, computers, cameras, watches and shoes. In order to support the evaluation of machine learning-based matching methods, the data is split into training, validation and test sets. For each product category, we provide training sets in four different sizes (2.000-70.000 pairs). Furthermore there are sets of ids for each training set for a possible validation split (stratified random draw) available. The test set for each product category consists of 1.100 product pairs. The labels of the test sets were manually checked while those of the training sets were derived using shared product identifiers from the Web via weak supervision. The data stems from the WDC Product Data Corpus for Large-Scale Product Matching - Version 2.0 which consists of 26 million product offers originating from 79 thousand websites.

DBLP Temporal is a dataset for temporal entity resolution, based on author profiles extracted from the Digital Bibliography and Library Project (DBLP).

WDC Products is an entity matching benchmark which provides for the systematic evaluation of matching systems along combinations of three dimensions while relying on real-word data. The three dimensions are i) amount of corner-cases ii) generalization to unseen entities, and iii) development set size WDC Products contains 11715 product offers describing in total 2162 product entities belonging to various product categories. Source: WDC Products: A Multi-Dimensional Entity Matching Benchmark

The MusicBrainz20K dataset for entity resolution and entity clustering is based on real records about songs from the MusicBrainz database. Each record is described with the following attributes: artist, title, album, year and length. The records have been modified with the DAPO [1] data generator. The generated dataset consists of five sources and approximately 20K records describing 10K unique song entities. It contains duplicates for 50% of the original records in two to five sources which are generated with a high degree of corruption to stress-test the entity resolution and clustering approaches. [1] Hildebrandt, Kai, et al. "Large-scale data pollution with Apache Spark." IEEE Transactions on Big Data 6.2 (2017): 396-411.

Hand-disambiguation of a sample of U.S. patents inventor mentions from PatentsView.org. Inventors we selected indirectly by sampling inventor mentions uniformly at random. This results in inventor sampled with probability proportional to their number of granted patents. The time period considered is from 1976 to December 31, 2021, corresponding to the disambiguation labeled "disamb_inventor_id_20211230" in PatentsView's bulk data downloads "g_persistent_inventor.tsv" file (https://patentsview.org/download/data-download-tables). That is, the benchmark disambiguation intends to contain all inventor mentions for the sampled inventors from that time period. Note that the benchmark disambiguation contains a few extraneous mentions to patents granted outside of that time period. These should be ignored for evaluation purposes. The methodology used for the hand-disambiguation is described in Binette et al. (2022) (https://arxiv.org/abs/2210.01230). We used one disambiguation of 200 inventors from Binette et al. (2022), as well as an additional disambiguation of 200 inventors provided by an additional staff member. The two disambiguations were reviewed and validated. However, they should be expected to contain errors due to the ambiguous nature of inventor disambiguation. Furthermore, given the use as the December 30, 2021, disambiguation from PatentsView as a starting point of the hand-labeling, a bias towards this disambiguation should be expected.

CEREC is a large scale corpus for entity resolution in email conversations. The corpus consists of 6001 email threads from the Enron Email Corpus containing 36,448 email messages and 60,383 entity coreference chains. The annotation is carried out as a two-step process with minimal manual effort.

Dataset for our paper Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky which includes 5000 enterprise tools and the corresponding dialogues generated using DiaFORGE UTC data engine. The dataset is generated with the data generation engine. The engine simulates a user agent and an assistant agent in a dialogue, where the user agent has a persona and the assistant agent has access to a set of tools. Detailed information about the data generation process can be found in the paper. Each entry in the dataset contains: - seed: Gold tool for the generated dialogue. - user_persona: Persona of the simulated user agent. - messages: List of messages in the dialogue, where each message is a dictionary containing: - distractor_tools: List of distractor tools that are not the gold tool but are relevant to the dialogue. These tools are used by user agent to generate utterances that are hard to disambiguate from the gold tool. - retrieved_tools: List of tools retrieved by the assistant agent and used by the assistant agent in order to ask clarifying questions to the user agent.

The dataset contains entities from IMDB, TheMovieDB and TheTVDB with goldstandard matches between the sources. Due to the licensing of IMDB we provide a script to build the IMDB part of the dataset yourself. The dataset contains a variety of entity types to match: persons, movies, series, episodes and companies.

PIZZA is a dataset for parsing pizza and drink orders, whose semantics cannot be captured by flat slots and intents. Source: PIZZA: A new benchmark for complex end-to-end task-oriented parsing Image Source: https://arxiv.org/pdf/2212.00265v1.pdf

WDC Block is a benchmark for comparing the performance of blocking methods that are used as part of entity resolution pipelines. Entity resolution aims to identify records in two datasets (A and B) that describe the same real-world entity. Since comparing all record pairs between two datasets can be computationally expensive, entity resolution is approached in two steps, blocking and matching. Blocking applies a computationally cheap method to remove non-matching record pairs and produces a smaller set of candidate record pairs reducing the workload of the matcher. During matching a more expensive pair-wise matcher produces a final set of matching record pairs. Existing benchmark datasets for blocking and matching are rather small with respect to the Cartesian product AxB for comparing all records and the vocabulary size. If blockers are evaluated only on these small datasets, effects resulting from a high number of records or from a large vocabulary size (large number of unique tokens that need to be indexed) may be missed. The Web Data Commons Block (WDC-Block) is a new blocking benchmark that provides much larger datasets and thus requires blockers that address these scalability challenges. WDC Block features a maximal Cartesian product of 200 billion pairs of product offers which were extracted form 3,259 e-shops. Additionally, we provide three development sets with different sizes (~1K pairs, ~5K pairs & ~20K pairs) to experiment with different amounts of training data for the blockers.

This dataset is used for user identity linkage across two online social networks in Chinese. It contains two popular Chinese social platforms: Sina Weibo\footnote{https://weibo.com} and Douban\footnote{https://www.douban.com}. Details: * 9,714 users and 117,218 relations in Weibo; 9,526 users and 120,245 relations in Douban; 1,397 pair of matched users. * Approximate power-law degree distribution and high aggregation coefficient. * Multiple text attributes available, including username, geographical location and recent (text) posts of the users. * Construction time: April 2020.