Translation

xP3 is a multilingual dataset for multitask prompted finetuning. It is a composite of supervised datasets in 46 languages with English and machine-translated prompts. Source: Crosslingual Generalization through Multitask Finetuning Image Source: https://arxiv.org/pdf/2211.01786v1.pdf

The Oxford Radar RobotCar Dataset is a radar extension to The Oxford RobotCar Dataset. It has been extended with data from a Navtech CTS350-X Millimetre-Wave FMCW radar and Dual Velodyne HDL-32E LIDARs with optimised ground truth radar odometry for 280 km of driving around Oxford, UK (in addition to all sensors in the original Oxford RobotCar Dataset). Source: The Oxford Radar RobotCar Dataset: A Radar Extension to the Oxford RobotCar Dataset

CVSS is a massively multilingual-to-English speech to speech translation (S2ST) corpus, covering sentence-level parallel S2ST pairs from 21 languages into English. CVSS is derived from the Common Voice speech corpus and the CoVoST 2 speech-to-text translation (ST) corpus, by synthesizing the translation text from CoVoST 2 into speech using state-of-the-art TTS systems

This is the dataset for the 2020 Duolingo shared task on Simultaneous Translation And Paraphrase for Language Education (STAPLE). Sentence prompts, along with automatic translations, and high-coverage sets of translation paraphrases weighted by user response are provided in 5 language pairs. Starter code for this task can be found here: github.com/duolingo/duolingo-sharedtask-2020/. More details on the data set and task are available at: sharedtask.duolingo.com

GigaST is a large-scale pseudo speech translation (ST) corpus. The corpus was created by translating the text in GigaSpeech, an English ASR corpus, into German and Chinese. The training set is translated by a strong machine translation system and the test set was translated by human. ST models trained with an addition of the corpus obtain new state-of-the-art results on the MuST-C English-German benchmark test set.

ACES a dataset consisting of 68 phenomena ranging from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. It can be used to evaluate a wide range of Machine Translation metrics.

Demetr is a diagnostic dataset with 31K English examples (translated from 10 source languages) for evaluating the sensitivity of MT evaluation metrics to 35 different linguistic perturbations spanning semantic, syntactic, and morphological error categories. Source: DEMETR: Diagnosing Evaluation Metrics for Translation Image Source: https://arxiv.org/pdf/2210.13746v1.pdf

PhoMT is a high-quality and large-scale Vietnamese-English parallel dataset of 3.02M sentence pairs for machine translation.

SpeechMatrix is a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. Source: SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations Image Source: (https://scontent-lhr8-2.xx.fbcdn.net/v/t39.8562-6/310002966_605149234737289_5204270723809834290_n.pdf?_nc_cat=102&ccb=1-7&_nc_sid=ad8a9d&_nc_ohc=FN2KnupyKI0AX90B5UO&_nc_ht=scontent-lhr8-2.xx&oh=00_AT9iFWHchGOnkzVTmwiYIDElIXSnwilSGhDwRQdFh99rlA&oe=63560915

FRMT is a dataset and evaluation benchmark for Few-shot Region-aware Machine Translation, a type of style-targeted translation. The dataset consists of human translations of a few thousand English Wikipedia sentences into regional variants of Portuguese and Mandarin. Source documents are selected to enable detailed analysis of phenomena of interest, including lexically distinct terms and distractor terms. Source:

A dataset that contains the description of an image or a section within the image in Hausa and its equivalent in English. Hausa, a Chadic language, is a member of the Afro-Asiatic language family. It is estimated that about 100 to 150 million people speak the language, with more than 80 million indigenous speakers. The dataset comprises 32,923 images and their descriptions that are divided into training, development, test, and challenge test set. The Hausa Visual Genome is the first dataset of its kind and can be used for Hausa-English machine translation, multi-modal research, and image description, among various other natural language processing and generation tasks.

MuLD (Multitask Long Document Benchmark) is a set of 6 NLP tasks where the inputs consist of at least 10,000 words. The benchmark covers a wide variety of task types including translation, summarization, question answering, and classification. Additionally there is a range of output lengths from a single word classification label all the way up to an output longer than the input text.

PETCI is a Parallel English Translation dataset of Chinese Idioms, collected from an idiom dictionary and Google and DeepL translation. PETCI contains 4,310 Chinese idioms with 29,936 English translations. These translations capture diverse translation errors and paraphrase strategies.

This dataset is parallel text for Bornholmsk and Danish.

The BWB corpus consists of Chinese novels translated by experts into English, and the annotated test set is designed to probe the ability of machine translation systems to model various discourse phenomena. Source: A Bilingual Parallel Corpus with Discourse Annotations Image Source: https://arxiv.org/pdf/2210.14667v1.pdf

The FairTranslate Dataset includes 2,418 sentence pairs, each centered around an occupation, designed to assess gender expression and translation in English-French contexts. Each English sentence appears in three gender variants (male, female, inclusive), allowing for direct counterfactual comparisons. This structure supports fairness evaluations and helps analyze how models handle grammatical gender, inclusive forms, and coreference resolution in translation. Each example in the dataset is annotated with rich metadata: - English: Sentence involving an occupation, designed to test both explicit and subtle cues of gender. - French: Ground-truth translation faithfully aligned with the intended gender variant. - Gender: Target gender for translation: male, female, or inclusive. - Ambiguity: Level of gender ambiguity in the English source sentence: - ambiguous: No explicit gender cues. - unambiguous: Clear pronouns or cues (e.g., “he”, “she”, “they”). - long unambiguous: Gender resolvable from distant context, testing long-range coreference. - Stereotype: Whether the occupation is male-stereotyped, female-stereotyped, or gender-balanced, based on real-world statistics from Statbel. - Occupation: A list of the three gendered French forms for each occupation, e.g., ["infirmier", "infirmière", "infirmier.ière"].

This is a super-parallel Bible corpus containing 1401 language labels (language_script pairs), meaning that for each verse, we have the translation in other languages. However, this results in a relatively low number of verses, with the current version supporting 103 verses per language.