Machine translation is the task of translating a sentence in a source language to a different target language.
Approaches for machine translation can range from rule-based to statistical to neural-based. More recently, encoder-decoder attention-based architectures like BERT have attained major improvements in machine translation.
One of the most popular datasets used to benchmark machine translation systems is the WMT family of datasets. Some of the most commonly used evaluation metrics for machine translation systems include BLEU, METEOR, NIST, and others.
<span style="color:grey; opacity: 0.6">( Image credit: Google seq2seq )</span>
데이터셋
출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.
BPersona-chat is an evaluation dataset based on the English multiturn chat corpus Persona-chat and the Japanese multiturn chat corpus JPersona-chat. Each chat was performed between two crowd workers assuming artificial personas. The speakers discuss a given personality trait, including but not limited to self-introduction, hobby, and others. (Notice that they are not translations of each other.) Chats are translated into Japanese/English by professional translators, a low-quality machine translation model A and a high-quality machine translation model B. Translations are evaluated by crowdworkers as either good or bad, depending on the correctness and coherence. Each chat is included in one .xlsx file with the following structure: person - the speaker on the current utterance, source - the utterance in the source language, translation - the translation in the target language, evaluation: is this a good translation? - the evaluation of the translation's quality, y - the current translation is a correct translation of the source utterance, n - the current translation is an erroneous translation of the source utterance.
Previous research for adapting a general neural machine translation (NMT) model into a specific domain usually neglects the diversity in translation within the same domain, which is a core problem for domain adaptation in real- world scenarios. One representative of such challenging scenarios is to deploy a translation system for a conference with a specific topic, e.g., global warming or coronavirus, where there are usually extremely less resources due to the limited schedule. To motivate wider investigation in such a scenario, we present a real-world fine-grained domain adaptation task in machine translation (FGraDA). The FGraDA dataset consists of Chinese-English translation task for four sub-domains of information technology: autonomous vehicles, AI education, real-time networks, and smart phone. Each sub-domain is equipped with a development set and test set for evaluation pur- poses. To be closer to reality, FGraDA does not employ any in-domain bilingual training data but provides bilingual dictionaries and wiki knowledge base, which can be easier obtained within a short time. We benchmark the fine-grained domain adaptation task and present in-depth analyses showing that there are still challenging problems to further improve the performance with heterogeneous resources.
HumanMT is a collection of human ratings and corrections of machine translations. It consists of two parts: The first part contains five-point and pairwise sentence-level ratings, the second part contains error markings and corrections. Details are described in the following. I. Sentence-level ratings This is a collection of five-point and pairwise ratings for 1000 German-English machine translations of TED talks (IWSLT 2014). The ratings were collected with the purpose of assessing machine translation quality rating reliability and learnability to improve a neural machine translation model with human reinforcement (see publications). II. Error markings and corrections This is a collection of word-level error markings and post-edits/corrections for 3120 English-German machine translated sentences of 30 selected TED talks (IWSLT 2017). Each sentence received either a correction or a marking of errors from human annotators. This data was collected with the purpose of comparing annotation cost and quality, and potential for downstream machine translation improvements between annotation modes (see publications). Source: HumanMT: Human Machine Translation Ratings
IgboNLP is a standard machine translation benchmark dataset for Igbo. It consists of 10,000 English-Igbo human-level quality sentence pairs mostly from the news domain. Source: Igbo-English Machine Translation: An Evaluation Benchmark
a parallel corpus of Sorani (ckb or Central Kurdish) and Kurmanji (kmr or Northern Kurdish) dialects of Kurdish along with English (eng). The parallel corpus contains three manually-aligned corpus in Sorani-Kurmanji, Sorani-English and Kurmanji-English in various formats, namely Translation Memory eXchange file format (.tmx), parallel annotated text useful for ParaConc and raw parallel texts (.txt). This corpus contains 12,327 translation pairs in the two major dialects of Kurdish, Sorani and Kurmanji. We also provide 1,797 and 650 translation pairs in English-Kurmanji and English-Sorani.
MENYO-20k is the first multi-domain parallel corpus with a special focus on clean orthography for Yorùbá--English with standardized train-test splits for benchmarking.
Dataset Description The dataset used in this study comprises bug reports extracted from the Visual Studio Code GitHub repository, specifically focusing on those labeled with the english-please tag. This label indicates that the original submission was written in a language other than English, providing a clear signal for multilingual content. The dataset spans a five-year period (March 2019--June 2024), ensuring a diverse representation of bug types, user environments, and technical contexts. ### Characteristics The dataset contains 1,381 multilingual bug reports, each consisting of: - The original bug report written in a non-English language. - A translated version in English. - Metadata such as issue number, creation date, labels, and status. - Categorization into functional, UI, and performance-related issues based on the content. ### Motivation & Summary This dataset is motivated by the need to improve multilingual bug tracking and translation evaluation. Given the increasing globalization of software development, developers and QA teams frequently encounter bug reports in languages they do not understand. By providing a structured corpus of translated bug reports, this dataset facilitates: - Comparative translation evaluation (e.g., ChatGPT vs AWS Translate vs DeepL). - Linguistic analysis of technical bug reporting across different languages. - Insights into common software issues encountered by diverse users. - Improving multilingual issue tracking through automated labeling and categorization. ### Potential Use Cases This dataset can be beneficial for various research and development applications, including: - Machine Translation Benchmarking: Evaluating the performance of translation models in a technical domain. - Natural Language Processing (NLP) Tasks: Training classifiers to categorize bug reports based on their content. - Software Engineering Research: Understanding trends in bug reporting, issue resolution, and localization challenges. - Automated Bug Triage: Developing AI-driven solutions for assigning and prioritizing bug reports in multilingual repositories.
PheMT is a phenomenon-wise dataset designed for evaluating the robustness of Japanese-English machine translation systems. The dataset is based on the MTNT dataset, with additional annotations of four linguistic phenomena common in UGC; Proper Noun, Abbreviated Noun, Colloquial Expression, and Variant Source: https://github.com/cl-tohoku/PheMT Image Source: Fujii et al
PolyNews is a multilingual parallel dataset containing news titles 833 language pairs, spanning in 64 languages and 17 scripts. PolyNewsParallel aims to provide an easily-accessible, unified and de-duplicated dataset that combines three disparate data sources. It can be used for machine translation or text retrieval in both high-resource and low-resource languages.
scb-mt-en-th-2020 is an English-Thai machine translation dataset with over 1 million segment pairs, curated from various sources, namely news, Wikipedia articles, SMS messages, task-based dialogs, web-crawled data and government documents. Source: scb-mt-en-th-2020: A Large English-Thai Parallel Corpus Image Source: Lowphansirikul et al
SimpEvalASSET is a dataset for learning learnable metrics using modern language models. It comprises of 12K human ratings on 2.4K simplifications of 24 systems, and SIMPEVAL_2022, a challenging simplification benchmark consisting of over 1K human ratings of 360 simplifications including generations from GPT-3.5. Source: LENS: A Learnable Evaluation Metric for Text Simplification Image Source: https://arxiv.org/pdf/2212.09739v1.pdf
The Medical Translation Task of WMT 2014 addresses the problem of domain-specific and genre-specific machine translation. The task is split into two subtasks: summary translation, focused on translation of sentences from summaries of medical articles, and query translation, focused on translation of queries entered by users into medical information search engines. Both subtasks included translation between English and Czech, German, and French, in both directions. Source: https://www.aclweb.org/anthology/W14-3302.pdf
News translation is a recurring WMT task. The test set is a collection of parallel corpora consisting of about 1500 English sentences translated into 5 languages (Czech, German, Finnish, French, Russian) and additional 1500 sentences from each of the 5 languages translated to English. The sentences are taken from newspaper articles for each language pair, except for French, where the test set was drawn from user-generated comments on the news articles (from Guardian and Le Monde). The translation was done by professional translators. The training data consists of parallel corpora to train translation models, monolingual corpora to train language models and development sets for tuning. Some training corpora were identical from WMT 2014 (Europarl, United Nations, French-English 10⁹ corpus, CzEng, Common Crawl, Russian-English parallel data provided by Yandex, Wikipedia Headlines provided by CMU) and some were update (News Commentary, monolingual news data). Additionally, the Finnish Europarl and Finnish-English Wikipedia Headline corpus were added. Source: https://paperswithcode.com/paper/findings-of-the-2016-conference-on-machine/ Image Source: httpshttps://www.aclweb.org/anthology/W15-3001.pdf
The Biomedical Translation Shared Task was first introduced at the First Conference of Machine Translation. The task aims to evaluate systems for the translation of biomedical titles and abstracts from scientific publications. The data includes three language pairs (English ↔ Portuguese, English ↔ Spanish, English ↔ French) and two sub-domains of biological sciences and health sciences. The training data consists mainly of the Scielo corpus, a parallel collection of scientific publications composed of either titles, abstracts or title and abstracts which were retrieved from the Scielo database. For the Scielo corpus, a parallel documents are provided for all language pairs in the two sub-domains, except for the English ↔ French, where only health was considered, as there were inadequate parallel documents available for biology in that pair. The training data was aligned using the GMA alignment tool. Additionally, a corpus of parallel titles from MEDLINEⓇ for all three language pairs were provided as well as monolingual documents for the four languages, retrieved from the Scielo database. These consist of documents in the Scielo database which have no corresponding document in another language. The test set consisted of 500 documents (title and abstract) for each of the two directions of each language pair. None of the test documents was included in the training data and there is no overlap of documents between the test sets for any language pair, translation direction and sub-domain. Source: http://www.statmt.org/wmt16/index.html Image Source: https://www.aclweb.org/anthology/W16-2301.pdf
The IT Translation Task is a shared task introduced in the First Conference on Machine Translation. Compared to WMT 2016 News, this task brought several novelties to WMT: * 4 out of the 7 langauges of the IT task are new in WMT, * adaptation to the IT domain with its specifics such as frequent named entities (mostly menu items, names of products and companies) and technical jargon, * adaptation to translation of answers in helpdesk service setting (many of the sentences are instructions with imperative verbs, which is very rare in the News translation task). The test set consisted of 1000 answers from the Batch 3 of the QTLeap Corpus. The in-domain training data contained 2000 answers from the Batches 1 and 2 and also localization files from several open-source projects (LibreOffice, KDE, VLC) and bilingual dictionaries of IT-related terms extracted from Wikipedia. The out-of-domain training data contained all the corpora from the WMT 2016 News, plus PaCo2-EuEn Basque-English corpus and SETimes with Bulgarian-English parallel sentences. “Constrained” systems were restricted to use only these training data provided by the organizers. The task was evaluated on the following language pairs: * English → Bulgarian * English → Czech * English → German * English → Spanish * English → Basque * English → Dutch * English → Portuguese Source: http://www.statmt.org/wmt16/index.html Image Source: Bojar et al
Current approaches to context-aware MT rely on a set of surface heuristics to translate pronouns, which break down when translations require real reasoning. We create a new template test set ContraCAT to assess the ability of Machine Translation to handle the specific steps necessary for successful pronoun translation.
We present sentence aligned parallel corpora across 10 Indian Languages - Hindi, Telugu, Tamil, Malayalam, Gujarati, Urdu, Bengali, Oriya, Marathi, Punjabi, and English - many of which are categorized as low resource. The corpora are compiled from online sources which have content shared across languages. The corpora presented significantly extends present resources that are either not large enough or are restricted to a specific domain (such as health). We also provide a separate test corpus compiled from an independent online source that can be independently used for validating the performance in 10 Indian languages. Alongside, we report on the methods of constructing such corpora using tools enabled by recent advances in machine translation and cross-lingual retrieval using deep neural network based methods.
WMT 2021 Ge'ez-Amharic is a Ge'ez-Amharic dataset prepared for NMT tasks of the 6th Workshop on NLP at Debre Berhan University, Ethiopia. The corpus has been collected from: * Ethiopian Orthodox Church old bible (from ethiopianorthodox.org), Anaphora, praise of St. Virgin Mary, praise of Lord Jesus and other Church's books. * Ge'ez teaching books, * Websites and other internet sources such as www.geez.org, www.debelo.org, The Dataset has about 15454 parallel Ge'ez and Amharic sentences for training, 1001 parallel sentences for testing and 1001 parallel sentences for validation.