Semantic Textual Similarity

General Language Understanding Evaluation (GLUE) benchmark is a collection of nine natural language understanding tasks, including single-sentence tasks CoLA and SST-2, similarity and paraphrasing tasks MRPC, STS-B and QQP, and natural language inference tasks MNLI, QNLI, RTE and WNLI. Source: Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models Image Source: https://gluebenchmark.com/

The IMDb Movie Reviews dataset is a binary sentiment analysis dataset consisting of 50,000 reviews from the Internet Movie Database (IMDb) labeled as positive or negative. The dataset contains an even number of positive and negative reviews. Only highly polarizing reviews are considered. A negative review has a score ≤ 4 out of 10, and a positive review has a score ≥ 7 out of 10. No more than 30 reviews are included per movie. The dataset contains additional unlabeled data. Source: http://nlpprogress.com/english/sentiment_analysis.html Image Source: Maas et al

This is an evaluation harness for the HumanEval problem solving dataset described in the paper "Evaluating Large Language Models Trained on Code". It used to measure functional correctness for synthesizing programs from docstrings. It consists of 164 original programming problems, assessing language comprehension, algorithms, and simple mathematics, with some comparable to simple software interview questions. Source: Evaluating Large Language Models Trained on Code Image Source: Evaluating Large Language Models Trained on Code

Microsoft Research Paraphrase Corpus (MRPC) is a corpus consists of 5,801 sentence pairs collected from newswire articles. Each pair is labelled if it is a paraphrase or not by human annotators. The whole set is divided into a training subset (4,076 sentence pairs of which 2,753 are paraphrases) and a test subset (1,725 pairs of which 1,147 are paraphrases). Source: Exploiting Semantic Annotations and Q-Learning for Constructing an Efficient Hierarchy/Graph Texts Organization Image Source: https://www.aclweb.org/anthology/I05-5002.pdf

The Sentences Involving Compositional Knowledge (SICK) dataset is a dataset for compositional distributional semantics. It includes a large number of sentence pairs that are rich in the lexical, syntactic and semantic phenomena. Each pair of sentences is annotated in two dimensions: relatedness and entailment. The relatedness score ranges from 1 to 5, and Pearson’s r is used for evaluation; the entailment relation is categorical, consisting of entailment, contradiction, and neutral. There are 4439 pairs in the train split, 495 in the trial split used for development and 4906 in the test split. The sentence pairs are generated from image and video caption datasets before being paired up using some algorithm. Source: Multi-Label Transfer Learning for Multi-Relational Semantic Similarity Image Source: https://www.researchgate.net/figure/Example-of-SICK-dataset-sentence-expansion-process-14_fig1_344863619

PAWS-X contains 23,659 human translated PAWS evaluation pairs and 296,406 machine translated training pairs in six typologically distinct languages: French, Spanish, German, Chinese, Japanese, and Korean. All translated pairs are sourced from examples in PAWS-Wiki. Source: Google Research Image Source: https://arxiv.org/pdf/1908.11828v1.pdf

SentEval is a toolkit for evaluating the quality of universal sentence representations. SentEval encompasses a variety of tasks, including binary and multi-class classification, natural language inference and sentence similarity. The set of tasks was selected based on what appears to be the community consensus regarding the appropriate evaluations for universal sentence representations. The toolkit comes with scripts to download and preprocess datasets, and an easy interface to evaluate sentence encoders. Source: SentEval: An Evaluation Toolkit for Universal Sentence Representations

Paraphrase Adversaries from Word Scrambling (PAWS) is a dataset contains 108,463 human-labeled and 656k noisily labeled pairs that feature the importance of modeling structure, context, and word order information for the problem of paraphrase identification. The dataset has two subsets, one based on Wikipedia and the other one based on the Quora Question Pairs (QQP) dataset. Source: PAWS

MTEB is a benchmark that spans 8 embedding tasks covering a total of 56 datasets and 112 languages. The 8 task types are Bitext mining, Classification, Clustering, Pair Classification, Reranking, Retrieval, Semantic Textual Similarity and Summarisation. The 56 datasets contain varying text lengths and they are grouped into three categories: Sentence to sentence, Paragraph to paragraph, and Sentence to paragraph. Check the latest leaderboards at HuggingFace.

CARER is an emotion dataset collected through noisy labels, annotated via distant supervision as in (Go et al., 2009). The subset of data provided here corresponds to the six emotions variant described in the paper. The six emotions are anger, fear, joy, love, sadness, and surprise. Source: CARER: Contextualized Affect Representations for Emotion Recognition

The Yelp Dataset is a valuable resource for academic research, teaching, and learning. It provides a rich collection of real-world data related to businesses, reviews, and user interactions. Here are the key details about the Yelp Dataset: Reviews: A whopping 6,990,280 reviews from users. Businesses: Information on 150,346 businesses. Pictures: A collection of 200,100 pictures. Metropolitan Areas: Data from 11 metropolitan areas. Tips: Over 908,915 tips provided by 1,987,897 users. Business Attributes: Details like hours, parking availability, and ambiance for more than 1.2 million businesses. Aggregated Check-ins: Historical check-in data for each of the 131,930 businesses.

WikiHop is a multi-hop question-answering dataset. The query of WikiHop is constructed with entities and relations from WikiData, while supporting documents are from WikiReading. A bipartite graph connecting entities and documents is first built and the answer for each query is located by traversal on this graph. Candidates that are type-consistent with the answer and share the same relation in query with the answer are included, resulting in a set of candidates. Thus, WikiHop is a multi-choice style reading comprehension data set. There are totally about 43K samples in training set, 5K samples in development set and 2.5K samples in test set. The test set is not provided. The task is to predict the correct answer given a query and multiple supporting documents. The dataset includes a masked variant, where all candidates and their mentions in the supporting documents are replaced by random but consistent placeholder tokens. Source: Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs Image Source: http://qangaroo.cs.ucl.ac.uk/

Quora Question Pairs (QQP) dataset consists of over 400,000 question pairs, and each question pair is annotated with a binary value indicating whether the two questions are paraphrase of each other. Source: Bilateral Multi-Perspective Matching for Natural Language Sentences

STS Benchmark comprises a selection of the English datasets used in the STS tasks organized in the context of SemEval between 2012 and 2017. The selection of datasets include text from image captions, news headlines and user forums. Source: STS Benchmark

EVALution dataset is evenly distributed among the three classes (hypernyms, co-hyponyms and random) and involves three types of parts of speech (noun, verb, adjective). The full dataset contains a total of 4,263 distinct terms consisting of 2,380 nouns, 958 verbs and 972 adjectives. Source: Network Features Based Co-hyponymy Detection Image Source: https://www.aclweb.org/anthology/W15-4208.pdf

Crisscrossed Captions (CxC) contains 247,315 human-labeled annotations including positive and negative associations between image pairs, caption pairs and image-caption pairs. Image source: Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO

Paraphrase and Semantic Similarity in Twitter (PIT) presents a constructed Twitter Paraphrase Corpus that contains 18,762 sentence pairs. Source: SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT)

Publicly available dataset of naturally occurring factual claims for the purpose of automatic claim verification. It is collected from 26 fact checking websites in English, paired with textual sources and rich metadata, and labelled for veracity by human expert journalists. Source: MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims

Paralex learns from a collection of 18 million question-paraphrase pairs scraped from WikiAnswers.

KorNLI is a Korean Natural Language Inference (NLI) dataset. The dataset is constructed by automatically translating the training sets of the SNLI, XNLI and MNLI datasets. To ensure translation quality, two professional translators with at least seven years of experience who specialize in academic papers/books as well as business contracts post-edited a half of the dataset each and cross-checked each other’s translation afterward. It contains 942,854 training examples translated automatically and 7,500 evaluation (development and test) examples translated manually Source: https://github.com/kakaobrain/KorNLUDatasets

PARANMT-50M is a dataset for training paraphrastic sentence embeddings. It consists of more than 50 million English-English sentential paraphrase pairs. Source: ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations

FLUE is a French Language Understanding Evaluation benchmark. It consists of 5 tasks: Text Classification, Paraphrasing, Natural Language Inference, Constituency Parsing and Part-of-Speech Tagging, and Word Sense Disambiguation.

JGLUE, Japanese General Language Understanding Evaluation, is built to measure the general NLU ability in Japanese.

SemEval 2014 is a collection of datasets used for the Semantic Evaluation (SemEval) workshop, an annual event that focuses on the evaluation and comparison of systems that can analyze diverse semantic phenomena in text. The datasets from SemEval 2014 are used for various tasks, including but not limited to: - Aspect-Based Sentiment Analysis (ABSA): This task is based on laptop and restaurant reviews. It involves identifying the aspects or features mentioned in a review and determining the sentiment expressed towards each aspect. - Text Classification: This task involves classifying text into predefined categories. Sub-tasks include text-scoring, natural language inference, and semantic-similarity-scoring.

Twitter News URL Corpus is a human-labeled paraphrase corpus to date of 51,524 sentence pairs and the first cross-domain benchmarking for automatic paraphrase identification. Source: A Continuously Growing Dataset of Sentential Paraphrases Image Source: https://arxiv.org/pdf/1708.00391v1.pdf

Finnish Paraphrase Corpus is a fully manually annotated paraphrase corpus for Finnish containing 53,572 paraphrase pairs harvested from alternative subtitles and news headings. Out of all paraphrase pairs in the corpus 98% are manually classified to be paraphrases at least in their given context, if not in all contexts.

SV-Ident comprises 4,248 sentences from social science publications in English and German. The data is the official data for the Shared Task: “Survey Variable Identification in Social Science Publications” (SV-Ident) 2022. Sentences are labeled with variables that are mentioned either explicitly or implicitly. The dataset supports the following tasks: - Variable Detection: identifying whether a sentence contains a variable mention or not. - Variable Disambiguation: identifying which variable from a given vocabulary is mentioned in a sentence.

PARADE contains paraphrases that overlap very little at the lexical and syntactic level but are semantically equivalent based on computer science domain knowledge, as well as non-paraphrases that overlap greatly at the lexical and syntactic level but are not semantically equivalent based on this domain knowledge. Source: PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge

This dataset can be used for semantic textual similarity tasks. It consists of duplicate and non-duplicate Github issues. It has 18565, 1547, and 1548 samples for train, validation, and test set, respectively.

This is a paraphrasing dataset created using the adversarial paradigm. A task was designed called the Adversarial Paraphrasing Task (APT) whose objective was to write sentences that mean the same as a given sentence but have as different syntactical and lexical properties as possible. As shown in the paper, this dataset can be used to measure the performance of paraphrase identifier models and train them. This dataset and the task associated with it (APT) can also be used to challenge neural networks to generate better adversarial paraphrases (the work has done this for T5-base), which will in turn help create better paraphrase identifiers.

This is a benchmark for neural paraphrase detection, to differentiate between original and machine-generated content. ####Training: 1,474,230 aligned paragraphs (98,282 original, 1,375,948 paraphrased with 3 models and 5 hyperparameter configurations each 98,282) extracted from 4,012 (English) Wikipedia articles. ####Testing: BERT-large (cased): arXiv - Original - 20,966; Paraphrased - 20,966; Theses - Original - 5,226; Paraphrased - 5,226; Wikipedia - Original - 39,241; Paraphrased - 39,241; RoBERTa-large (cased): arXiv - Original - 20,966; Paraphrased - 20,966; Theses - Original - 5,226; Paraphrased - 5,226; Wikipedia - Original - 39,241; Paraphrased - 39,241; Longformer-large (uncased): arXiv - Original - 20,966; Paraphrased - 20,966; Theses - Original - 5,226; Paraphrased - 5,226; Wikipedia - Original - 39,241; Paraphrased - 39,241;

For more details see https://huggingface.co/datasets/jpwahle/autoregressive-paraphrase-dataset

A dataset of sentence pairs annotated following the formalization. Source: Interpretable Semantic Textual Similarity: Finding and explaining differences between sentences

This dataset is used to train and evaluate models for the detection of machine-paraphrased text. The training set consists of 200,767 paragraphs (98,282 original, 102,485 paraphrased) extracted from 8,024 Wikipedia (English) articles (4,012 original, 4,012 paraphrased using the SpinBot API). The test set is divided into 3 subsets: one created from preprints of research papers on arXiv, one from graduation theses, and one from Wikipedia articles. Additionally, different marchine-paraphrasing methods were used. Test sets: SpinBot: arXiv - Original - 20,966; Spun - 20,867 Theses - Original - 5,226; Spun - 3,463 Wikipedia - Original - 39,241; Spun - 40,729 SpinnerChief-4W: arXiv - Original - 20,966; Spun - 21,671 Theses - Original - 2,379; Spun - 2,941 Wikipedia - Original - 39,241; Spun - 39,618 SpinnerChief-2W: arXiv - Original - 20,966; Spun - 21,719 Theses - Original - 2,379; Spun - 2,941 Wikipedia - Original - 39,241; Spun - 39,697

Translated SNLI Dataset in Marathi A translated version of the SNLI dataset in Marathi, designed for Semantic Textual Similarity (STS) tasks. The translations were generated using the model aryaumesh/english-to-marathi. #### Translation Quality Metrics: - BERT Precision: 88.57% - BERT Recall: 88.60% - BERT F1: 88.58% - TransQuest Score: 0.72

The data originate from the journalistic domain in the Czech language. We describe the process of collecting and annotating the data in detail. The dataset contains 138,556 human annotations divided into train and test sets. In total, 485 journalism students participated in the creation process. To increase the reliability of the test set, we compute the annotation as an average of 9 individual annotations. We evaluate the quality of the dataset by measuring inter and intra annotation annotators' agreements. Beside agreement numbers, we provide detailed statistics of the collected dataset. We conclude our paper with a baseline experiment of building a system for predicting the semantic similarity of sentences. Due to the massive number of training annotations (116 956), the model can perform significantly better than an average annotator (0,92 versus 0,86 of Person's correlation coefficients). See https://arxiv.org/abs/2108.08708