Word Embeddings

ConceptNet is a knowledge graph that connects words and phrases of natural language with labeled edges. Its knowledge is collected from many sources that include expert-created resources, crowd-sourcing, and games with a purpose. It is designed to represent the general knowledge involved in understanding language, improving natural language applications by allowing the application to better understand the meanings behind the words people use. Source: ConceptNet 5.5: An Open Multilingual Graph of General Knowledge Image Source: Speer et al

FrameNet is a linguistic knowledge graph containing information about lexical and predicate argument semantics of the English language. FrameNet contains two distinct entity classes: frames and lexical units, where a frame is a meaning and a lexical unit is a single meaning for a word. Source: Retrofitting Distributional Embeddings to Knowledge Graphswith Functional Relations

BookCorpus is a large collection of free novel books written by unpublished authors, which contains 11,038 books (around 74M sentences and 1G words) of 16 different sub-genres (e.g., Romance, Historical, Adventure, etc.). Source: Temporal Event Knowledge Acquisition via Identifying Narratives

WiC is a benchmark for the evaluation of context-sensitive word embeddings. WiC is framed as a binary classification task. Each instance in WiC has a target word w, either a verb or a noun, for which two contexts are provided. Each of these contexts triggers a specific meaning of w. The task is to identify if the occurrences of w in the two contexts correspond to the same meaning or not. In fact, the dataset can also be viewed as an application of Word Sense Disambiguation in practise. Source: WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations

BioASQ is a question answering dataset. Instances in the BioASQ dataset are composed of a question (Q), human-annotated answers (A), and the relevant contexts (C) (also called snippets). Source: Transferability of Natural Language Inference to Biomedical Question Answering Image Source: http://participants-area.bioasq.org/datasets/

The One Billion Word dataset is a dataset for language modeling. The training/held-out data was produced from the WMT 2011 News Crawl data using a combination of Bash shell and Perl scripts.

WinoBias contains 3,160 sentences, split equally for development and test, created by researchers familiar with the project. Sentences were created to follow two prototypical templates but annotators were encouraged to come up with scenarios where entities could be interacting in plausible ways. Templates were selected to be challenging and designed to cover cases requiring semantics and syntax separately. Source: WinoBias Image Source: https://uclanlp.github.io/corefBias/overview

PadChest is a labeled large-scale, high resolution chest x-ray dataset for the automated exploration of medical images along with their associated reports. This dataset includes more than 160,000 images obtained from 67,000 patients that were interpreted and reported by radiologists at Hospital San Juan Hospital (Spain) from 2009 to 2017, covering six different position views and additional information on image acquisition and patient demography. The reports were labeled with 174 different radiographic findings, 19 differential diagnoses and 104 anatomic locations organized as a hierarchical taxonomy and mapped onto standard Unified Medical Language System (UMLS) terminology. Of these reports, 27% were manually annotated by trained physicians and the remaining set was labeled using a supervised method based on a recurrent neural network with attention mechanisms. The labels generated were then validated in an independent test set achieving a 0.93 Micro-F1 score.

WikiMatrix is a dataset of parallel sentences in the textual content of Wikipedia for all possible language pairs. The mined data consists of: - 85 different languages, 1620 language pairs - 134M parallel sentences, out of which 34M are aligned with English Source: WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia

WikiANN, also known as PAN-X, is a multilingual named entity recognition dataset. It consists of Wikipedia articles that have been annotated with LOC (location), PER (person), and ORG (organization) tags in the IOB2 format¹². This dataset serves as a valuable resource for training and evaluating named entity recognition models across various languages. For instance, it includes information about notable individuals, places, and organizations mentioned in Wikipedia articles. Researchers and practitioners can use WikiANN to develop and improve natural language processing systems that identify and classify named entities in text. (1) wikiann · Datasets at Hugging Face. https://huggingface.co/datasets/wikiann. (2) wikiann | TensorFlow Datasets. https://tensorflow.google.cn/datasets/catalog/wikiann. (3) wikiann · Datasets at Hugging Face. https://huggingface.co/datasets/wikiann/viewer/en. (4) WikiAnn Dataset | Papers With Code. https://paperswithcode.com/dataset/wikiann-1.

CELEX database comprises three different searchable lexical databases, Dutch, English and German. The lexical data contained in each database is divided into five categories: orthography, phonology, morphology, syntax (word class) and word frequency. Source: Polysemy and Brevity versus Frequency in Language Image Source: https://www.aclweb.org/anthology/W17-7619.pdf

Over a period of many years during the 1990s, a large group of psychologists all over the world collected data in the ISEAR project, directed by Klaus R. Scherer and Harald Wallbott. Student respondents, both psychologists and non-psychologists, were asked to report situations in which they had experienced all of 7 major emotions (joy, fear, anger, sadness, disgust, shame, and guilt). In each case, the questions covered the way they had appraised the situation and how they reacted. The final data set thus contained reports on seven emotions each by close to 3000 respondents in 37 countries on all 5 continents. Source: https://www.unige.ch/cisa/research/materials-and-online-research/research-material/ Image Source: https://www.unige.ch/cisa/research/materials-and-online-research/research-material/

PanLex translates words in thousands of languages. Its database is panlingual (emphasizes coverage of every language) and lexical (focuses on words, not sentences). Source: PANLEX Image Source: http://www.lrec-conf.org/proceedings/lrec2014/pdf/1029_Paper.pdf

EVALution dataset is evenly distributed among the three classes (hypernyms, co-hyponyms and random) and involves three types of parts of speech (noun, verb, adjective). The full dataset contains a total of 4,263 distinct terms consisting of 2,380 nouns, 958 verbs and 972 adjectives. Source: Network Features Based Co-hyponymy Detection Image Source: https://www.aclweb.org/anthology/W15-4208.pdf

There are now many computer programs for automatically determining the sense of a word in context (Word Sense Disambiguation or WSD). The purpose of SENSEVAL is to evaluate the strengths and weaknesses of such programs with respect to different words, different varieties of language, and different languages. Source: SENSEVAL Image Source: http://www.itri.brighton.ac.uk/events/senseval/

The first parallel corpus composed from United Nations documents published by the original data creator. The parallel corpus presented consists of manually translated UN documents from the last 25 years (1990 to 2014) for the six official UN languages, Arabic, Chinese, English, French, Russian, and Spanish. Source: The United Nations Parallel Corpus v1.0

An expert-annotated word similarity dataset which provides a highly reliable, yet challenging, benchmark for rare word representation techniques. Source: Card-660: Cambridge Rare Word Dataset - a Reliable Benchmark for Infrequent Word Representation Models

Polyglot-NER builds massive multilingual annotators with minimal human expertise and intervention. Source: POLYGLOT-NER: Massive Multilingual Named Entity Recognition

The IndoSum dataset is a benchmark dataset for Indonesian text summarization. The dataset consists of news articles and manually constructed summaries. Source: https://github.com/kata-ai/indosum

Consists of 2,864 videos each with a label from 25 different classes corresponding to an event unfolding 5 seconds. The ERA dataset is designed to have a significant intra-class variation and inter-class similarity and captures dynamic events in various circumstances and at dramatically various scales. Source: ERA: A Dataset and Deep Learning Benchmark for Event Recognition in Aerial Videos

SEND (Stanford Emotional Narratives Dataset) is a set of rich, multimodal videos of self-paced, unscripted emotional narratives, annotated for emotional valence over time. The complex narratives and naturalistic expressions in this dataset provide a challenging test for contemporary time-series emotion recognition models. Source: Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset

A new challenging dataset that can be used for many pattern recognition tasks. Source: Judging a Book By its Cover

Introduces three datasets of expressing hate, commonly used topics, and opinions for hate speech detection, document classification, and sentiment analysis, respectively. Source: Classification Benchmarks for Under-resourced Bengali Language based on Multichannel Convolutional-LSTM Network

SemEval 2014 is a collection of datasets used for the Semantic Evaluation (SemEval) workshop, an annual event that focuses on the evaluation and comparison of systems that can analyze diverse semantic phenomena in text. The datasets from SemEval 2014 are used for various tasks, including but not limited to: - Aspect-Based Sentiment Analysis (ABSA): This task is based on laptop and restaurant reviews. It involves identifying the aspects or features mentioned in a review and determining the sentiment expressed towards each aspect. - Text Classification: This task involves classifying text into predefined categories. Sub-tasks include text-scoring, natural language inference, and semantic-similarity-scoring.

WikiNEuRal is a high-quality automatically-generated dataset for Multilingual Named Entity Recognition.

The WordNet Language Model Probing (WNLaMPro) dataset consists of relations between keywords and words. It contains 4 different kinds of relations: Antonym, Hypernym, Cohyponym and Corruption. Source: https://arxiv.org/pdf/1904.06707.pdf

A large-scale evaluation set that provides human ratings for the plausibility of 10,000 SP pairs over five SP relations, covering 2,500 most frequent verbs, nouns, and adjectives in American English. Source: SP-10K: A Large-scale Evaluation Set for Selectional Preference Acquisition

In this paper, we present AnlamVer, which is a semantic model evaluation dataset for Turkish designed to evaluate word similarity and word relatedness tasks while discriminating those two relations from each other. Our dataset consists of 500 word-pairs annotated by 12 human subjects, and each pair has two distinct scores for similarity and relatedness. Word-pairs are selected to enable the evaluation of distributional semantic models by multiple attributes of words and word-pair relations such as frequency, morphology, concreteness and relation types (e.g., synonymy, antonymy). Our aim is to provide insights to semantic model researchers by evaluating models in multiple attributes. We balance dataset word-pairs by their frequencies to evaluate the robustness of semantic models concerning out-of-vocabulary and rare words problems, which are caused by the rich derivational and inflectional morphology of the Turkish language. (from the original abstract of the dataset paper)

Some Like it Hoax is a fake news detection dataset consisting of 15,500 Facebook posts and 909,236 users. Source: Some Like it Hoax: Automated Fake News Detection in Social Networks

The WikiSem500 dataset contains around 500 per-language cluster groups for English, Spanish, German, Chinese, and Japanese (a total of 13,314 test cases). Source: https://arxiv.org/abs/1611.01547

Dataset Summary CANNOT is a dataset that focuses on negated textual pairs. It currently contains 77,376 samples, of which roughly of them are negated pairs of sentences, and the other half are not (they are paraphrased versions of each other). The most frequent negation that appears in the dataset is verbal negation (e.g., will → won't), although it also contains pairs with antonyms (cold → hot). <br> ## Languages CANNOT includes exclusively texts in English. <br> ## Dataset Structure The dataset is given as a .tsv file with the following structure: | premise | hypothesis | label | |:------------|:---------------------------------------------------|:-----:| | A sentence. | An equivalent, non-negated sentence (paraphrased). | 0 | | A sentence. | The sentence negated. | 1 | The dataset can be easily loaded into a Pandas DataFrame by running: Python import pandas as pd dataset = pd.read_csv('negation_dataset_v1.0.tsv', sep='\t') <br> ## Dataset Creation The dataset has been created by cleaning up and merging the following datasets: 1. Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation (see datasets/nan-nli). 2. GLUE Diagnostic Dataset (see datasets/glue-diagnostic). 3. Automated Fact-Checking of Claims from Wikipedia (see datasets/wikifactcheck-english). 4. From Group to Individual Labels Using Deep Features (see datasets/sentiment-labelled-sentences). In this case, the negated sentences were obtained by using the Python module negate. 5. It Is Not Easy To Detect Paraphrases: Analysing Semantic Similarity With Antonyms and Negation Using the New SemAntoNeg Benchmark (see datasets/antonym-substitution). Once processed, the number of remaining samples in each of the datasets above are: | Dataset | Samples | |:--------------------------------------------------------------------------|-----------:| | Not another Negation Benchmark | 118 | | GLUE Diagnostic Dataset | 154 | | Automated Fact-Checking of Claims from Wikipedia | 14,970 | | From Group to Individual Labels Using Deep Features | 2,110 | | It Is Not Easy To Detect Paraphrases | 8,597 | | <div align="right"><b>Total</b></div> | 25,949 | Additionally, for each of the negated samples, another pair of non-negated sentences has been added by paraphrasing them with the pre-trained model 🤗tuner007/pegasus_paraphrase. Finally, the swapped version of each pair (premise ⇋ hypothesis) has also been included, and any duplicates have been removed. With this, the number of premises/hypothesis in the CANNOT dataset that appear in the original datasets are: | <div align="left"><b>Dataset</b></div> | <div align="center"><b>Sentences</b></div> | |:--------------------------------------------------------------------------|----------------------:| | Not another Negation Benchmark | 552     (0.36 %) | | GLUE Diagnostic Dataset | 586     (0.38 %) | | Automated Fact-Checking of Claims from Wikipedia | 89,728   (59.98 %) | | From Group to Individual Labels Using Deep Features | 12,626     (8.16 %) | | It Is Not Easy To Detect Paraphrases | 17,198   (11.11 %) | | <div align="right"><b>Total</b></div> | 120,690   (77.99 %) | The percentages above are in relation to the total number of premises and hypothesis in the CANNOT dataset. The remaining 22.01 % (34,062 sentences) are the novel premises/hypothesis added through paraphrase and rule-based negation. <br> ## Additional Information <br> ### Licensing Information The CANNOT dataset is released under CC BY-SA 4.0. <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> <img alt="Creative Commons License" width="100px" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png"/> </a> <br> ### Citation Please cite our INLG 2023 paper, if you use our dataset. BibTeX: bibtex @misc{anschütz2023correct, title={This is not correct! Negation-aware Evaluation of Language Generation Systems}, author={Miriam Anschütz and Diego Miguel Lozano and Georg Groh}, year={2023}, eprint={2307.13989}, archivePrefix={arXiv}, primaryClass={cs.CL} } <br> ### Contributions Contributions to the dataset can be submitted through the project repository.

The dataset contains the main components of the news articles published online by the newspaper named <a href="https://gazzettadimodena.gelocal.it/modena">Gazzetta di Modena</a>: url of the web page, title, sub-title, text, date of publication, crime category assigned to each news article by the author. The news articles are written in Italian and describe 11 types of crime events occurred in the province of Modena between the end of 2011 and 2021. Moreover, the dataset includes data derived from the abovementioned components thanks to the application of Natural Language Processing techniques. Some examples are the place of the crime event occurrence (municipality, area, address and GPS coordinates), the date of the occurrence, and the type of the crime events described in the news article obtained by an automatic categorization of the text. In the end, news articles describing the same crime events (duplciates) are detected by calculating the document similarity. Now, we are working on the application of question answering to extract the 5W+1H and we plan to extend the current dataset with the obtained data. Other researchers can employ the dataset to apply other algorithms of text categorization and duplicate detection and compare their results with the benchmark. The dataset can be useful for several scopes, e.g., geo-localization of the events, text summarization, crime analysis, crime prediction, community detection, topic modeling.

The dataset consists of the features associated with 402 5-second sound samples. The 402 sounds range from easily identifiable everyday sounds to intentionally obscured artificial ones. The dataset aims to lower the barrier for the study of aural phenomenology as the largest available audio dataset to include an analysis of causal attribution. Each sample has been annotated with crowd-sourced descriptions, as well as familiarity, imageability, arousal, and valence ratings. Source: https://github.com/mitmedialab/HCU400

The IndicNLP corpus is a large-scale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. Source: https://arxiv.org/abs/2005.00085

Contains more than 6500 words semantically grouped under 110 categories. Source: Semantic Structure and Interpretability of Word Embeddings

Chinese Gigaword corpus consists of 2.2M of headline-document pairs of news stories covering over 284 months from two Chinese newspapers, namely the Xinhua News Agency of China (XIN) and the Central News Agency of Taiwan (CNA). Source: Order-Preserving Abstractive Summarization for Spoken Content Based on Connectionist Temporal Classification Image Source: https://catalog.ldc.upenn.edu/desc/addenda/LDC2011T13.jpg