Word Embeddings

The Climate Change Claims dataset for generating fact checking summaries contains claims broadly related to climate change and global warming from climatefeedback.org. It contains 1k documents from 104 different claims from 97 different domains. Source: https://arxiv.org/abs/2010.08570

This dataset contains information about Japanese word similarity including rare words. The dataset is constructed following the Stanford Rare Word Similarity Dataset. 10 annotators annotated word pairs with 11 levels of similarity. Source: https://github.com/tmu-nlp/JapaneseWordSimilarityDataset

The MUSE dataset contains bilingual dictionaries for 110 pairs of languages. For each language pair, the training seed dictionaries contain approximately 5000 word pairs while the evaluation sets contain 1500 word pairs. Source: Filtered Inner Product Projection for Multilingual Embedding Alignment Image Source: https://github.com/facebookresearch/MUSE

The pioNER corpus provides gold-standard and automatically generated named-entity datasets for the Armenian language. The automatically generated corpus is generated from Wikipedia. The gold-standard set is a collection of over 250 news articles from iLur.am with manual named-entity annotation. It includes sentences from political, sports, local and world news, and is comparable in size with the test sets of other languages. Source: https://github.com/ispras-texterra/pioner

SART is a collection of three datasets for Similarity, Analogies and Relatedness for the Tatar language. The three subsets are: * Similarity dataset - 202 pairs of words along with averaged human scores of similarity degree between the words (in 0-to-10 scale). For example, "йорт, бина, 7.69". * Relatedness dataset - 252 pairs of words along with averaged human scores of relatedness degree between the words. For example, "урам, балалар, 5.38". * Analogies dataset - set of analytical questions of the form A:B::C:D, meaning A to B as C to D, and D is to be predicted. For example, "Әнкара Төркия Париж Франция". Contains 34 categories, and in total 30 144 questions. Source: https://github.com/tat-nlp/SART

Collects a huge number of job descriptions from Dice.com - one of the most popular career website about Tech jobs in USA. From these job descriptions, skills are extracted for each one by using skills dictionary. Now, the dataset is presented by a list of collections of skills based on job descriptions. After crawling, there are a total of 5GB with more than 1,400,000 job descriptions. From these data, skills are extracted and performed as a list of skills in the same context, the context here includes skills in the same job description. Source: Skill2vec: Machine Learning Approach for Determining the Relevant Skills from Job Description

We provide a Mikolov-style word-analogy evaluation set specifically for Bangla, with a sample size of 16678, as well as a translated and curated version of the Mikolov dataset, which contains 10594 samples for cross-lingual research.

Collects posts (and their reactions) from Facebook pages of large supermarket chains. Source: Social Emotion Mining Techniques for Facebook Posts Reaction Prediction

A distant supervision dataset by linking the entire English ClueWeb09 corpus to Freebase. Source: Global Textual Relation Embedding for Relational Understanding

iLur News Texts is a dataset of over 12000 news articles from iLur.am, categorized into 7 classes: sport, politics, weather, economy, accidents, art, society. The articles are split into train (2242k tokens) and test sets (425k tokens). Source: iLur News Texts

A dataset of sentence pairs annotated following the formalization. Source: Interpretable Semantic Textual Similarity: Finding and explaining differences between sentences

A new word analogy task dataset for Indonesian. Source: KaWAT: A Word Analogy Task Dataset for Indonesian

Word embedding is a modern distributed word representations approach widely used in many natural language processing tasks. Converting the vocabulary in a legal document into a word embedding model facilitates subjecting legal documents to machine learning, deep learning, and other algorithms and subsequently performing the downstream tasks of natural language processing vis-à-vis, for instance, document classification, contract review, and machine translation. The most common and practical approach of accuracy evaluation with the word embedding model uses a benchmark set with linguistic rules or the relationship between words to perform analogy reasoning via algebraic calculation. This paper proposes establishing a 1,134 Legal Analogical Reasoning Questions Set (LARQS) from the 2,388 Chinese Codex corpus using five kinds of legal relations, which are then used to evaluate the accuracy of the Chinese word embedding model. Moreover, we discovered that legal relations might be ubiquitous in the word embedding model.

McQueen dataset contains 15k visual conversations and over 80k queries where each one is associated with a fully-specified rewrite version. In addition, for entities appearing in the rewrite, the corresponding image box annotation is provided. Source: McQueen: a Benchmark for Multimodal Conversational Query Rewrite Image Source: https://arxiv.org/pdf/2210.12775v1.pdf

Includes co-referent name string pairs along with their similarities.

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

This is a dataset for detection fake death hoaxes. It consists of of death reports collected from Twitter between 1st January, 2012 and 31st December, 2014. It was collected by tracking the keyword 'RIP', and matching those tweets in which a name is mentioned next to RIP. Matching names were identified by using Wikidata as a database of names. The dataset contains 4,007 death reports, of which 2,301 are real deaths, 1,092 are commemorations and 614 are fake deaths. Source: Early Detection of Social Media Hoaxes at Scale

Urban Dict spelling variant is a variant spelling dataset for use of NLP research in the informal domain. It consists of around 25k variant spelling pairs form UrbanDictionary. Source: https://arxiv.org/abs/1911.04669 Image Source: https://arxiv.org/pdf/1911.04669.pdf

We provide a Mikolov-style word-analogy evaluation set specifically for Bangla, with a sample size of 16678, as well as a translated and curated version of the Mikolov dataset, which contains 10594 samples for cross-lingual research.