Sentence Embeddings

WikiMatrix is a dataset of parallel sentences in the textual content of Wikipedia for all possible language pairs. The mined data consists of: - 85 different languages, 1620 language pairs - 134M parallel sentences, out of which 34M are aligned with English Source: WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia

Contains 51,583 descriptions of 11,046 objects from 800 ScanNet scenes. ScanRefer is the first large-scale effort to perform object localization via natural language expression directly in 3D. Source: ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language

Sentence Compression is a dataset where the syntactic trees of the compressions are subtrees of their uncompressed counterparts, and hence where supervised systems which require a structural alignment between the input and output can be successfully trained. Source: Overcoming the Lack of Parallel Data in Sentence Compression

For understanding multimodal language used in expressing humor. Source: UR-FUNNY: A Multimodal Language Dataset for Understanding Humor

The BIOSSES data set comprises total 100 sentence pairs all of which were selected from the "TAC2 Biomedical Summarization Track Training Data Set" . The sentence pairs were evaluated by five different human experts that judged their similarity and gave scores in a range [0-4]. Our guideline was prepared based on SemEval 2012 Task 6 Guideline. Image source: BIOSSES

Tatoeba is a free collection of example sentences with translations geared towards foreign language learners. It is available in more than 400 languages. Its name comes from the Japanese phrase “tatoeba” (例えば), meaning “for example”. It is written and maintained by a community of volunteers through a model of open collaboration. Individual contributors are known as Tatoebans.

Paraphrase and Semantic Similarity in Twitter (PIT) presents a constructed Twitter Paraphrase Corpus that contains 18,762 sentence pairs. Source: SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT)

Opusparcus is a paraphrase corpus for six European languages: German, English, Finnish, French, Russian, and Swedish. The paraphrases are extracted from the OpenSubtitles2016 corpus, which contains subtitles from movies and TV shows. For each target language, the Opusparcus data have been partitioned into three types of data sets: training, development and test sets. The training sets are large, consisting of millions of sentence pairs, and have been compiled automatically, with the help of probabilistic ranking functions. The development and test sets consist of sentence pairs that have been annotated manually; each set contains approximately 1000 sentence pairs that have been verified to be acceptable paraphrases by two annotators. Source: Opusparcus

OPUS is a growing collection of translated texts from the web. In the OPUS project we try to convert and align free online data, to add linguistic annotation, and to provide the community with a publicly available parallel corpus. OPUS is based on open source products and the corpus is also delivered as an open content package. We used several tools to compile the current collection. All pre-processing is done automatically. No manual corrections have been carried out.

The Video2GIF dataset contains over 100,000 pairs of GIFs and their source videos. The GIFs were collected from two popular GIF websites (makeagif.com, gifsoup.com) and the corresponding source videos were collected from YouTube in Summer 2015. IDs and URLs of the GIFs and the videos are provided, along with temporal alignment of GIF segments to their source videos. The dataset shall be used to evaluate GIF creation and video highlight techniques. In addition to the 100K GIF-video pairs, the dataset contains 357 pairs of GIFs and their source videos as the test set. The 357 videos come with a Creative Commons CC-BY license, which allowed the authors to redistribute the material with appropriate credit to make the results on test set reproducible even when some of the videos become unavailable. Source: Video2GIF

The Discovery datasets consists of adjacent sentence pairs (s1,s2) with a discourse marker (y) that occurred at the beginning of s2. They were extracted from the depcc web corpus. Markers prediction can be used in order to train a sentence encoders. Discourse markers can be considered as noisy labels for various semantic tasks, such as entailment (y=therefore), subjectivity analysis (y=personally) or sentiment analysis (y=sadly), similarity (y=similarly), typicality, (y=curiously) ... The specificity of this dataset is the diversity of the markers, since previously used data used only ~10 imbalanced classes. The author of the dataset provide: - a list of the 174 discourse markers - a Base version of the dataset with 1.74 million pairs (10k examples per marker) - a Big version with 3.4 million pairs - a Hard version with 1.74 million pairs where the connective couldn't be predicted with a fastText linear model Source: GitHub

SemEval 2014 is a collection of datasets used for the Semantic Evaluation (SemEval) workshop, an annual event that focuses on the evaluation and comparison of systems that can analyze diverse semantic phenomena in text. The datasets from SemEval 2014 are used for various tasks, including but not limited to: - Aspect-Based Sentiment Analysis (ABSA): This task is based on laptop and restaurant reviews. It involves identifying the aspects or features mentioned in a review and determining the sentiment expressed towards each aspect. - Text Classification: This task involves classifying text into predefined categories. Sub-tasks include text-scoring, natural language inference, and semantic-similarity-scoring.

GeoCoV19 is a large-scale Twitter dataset containing more than 524 million multilingual tweets. The dataset contains around 378K geotagged tweets and 5.4 million tweets with Place information. The annotations include toponyms from the user location field and tweet content and resolve them to geolocations such as country, state, or city level. In this case, 297 million tweets are annotated with geolocation using the user location field and 452 million tweets using tweet content.

AuxAD is a a distantly supervised dataset for acronym disambiguation. Source: https://github.com/PrimerAI/sdu-data

AuxAI is a distantly supervised dataset for acronym identification. Source: https://github.com/PrimerAI/sdu-data

• An evaluation test bed for assessing the robustness of sentence embedding models against user-informed misinformation edits. - A dataset containing perturbed and unperturbed claim pairs used for improving embedding model robustness through knowledge distillation.

COSTRA 1.0 is a dataset of complex sentence transformations. The dataset is intended for the study of sentence-level embeddings beyond simple word alternations or standard paraphrasing. The first version of the dataset is limited to sentences in Czech but the construction method is universal and the authors plan to use it also for other languages. The dataset consist of 4,262 unique sentences with average length of 10 words, illustrating 15 types of modifications such as simplification, generalization, or formal and informal language variation. Source: COSTRA 1.0: A Dataset of Complex Sentence Transformations

Composed of judgments from the French Court of cassation and their corresponding summaries. Source: STRASS: A Light and Effective Method for Extractive Summarization Based on Sentence Embeddings

A distant supervision dataset by linking the entire English ClueWeb09 corpus to Freebase. Source: Global Textual Relation Embedding for Relational Understanding