Unsupervised Machine Translation

WMT 2014 is a collection of datasets used in shared tasks of the Ninth Workshop on Statistical Machine Translation. The workshop featured four tasks: * a news translation task, * a quality estimation task, * a metrics task, * a medical text translation task. Source: https://www.aclweb.org/anthology/W14-3302.pdf

WMT 2016 is a collection of datasets used in shared tasks of the First Conference on Machine Translation. The conference builds on ten previous Workshops on statistical Machine Translation. The conference featured ten shared tasks: - a news translation task, - an IT domain translation task, - a biomedical translation task, - an automatic post-editing task, - a metrics task (assess MT quality given reference translation). - a quality estimation task (assess MT quality without access to any reference), - a tuning task (optimize a given MT system), - a pronoun translation task, - a bilingual document alignment task, - a multimodal translation task. Source: http://www.statmt.org/wmt16/index.html

CCMatrix uses ten snapshots of a curated common crawl corpus (Wenzek et al., 2019) totalling 32.7 billion unique sentences. Source: CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB

News translation is a recurring WMT task. The test set is a collection of parallel corpora consisting of about 1500 English sentences translated into 5 languages (Czech, German, Finnish, Romanian, Russian, Turkish) and additional 1500 sentences from each of the 5 languages translated to English. For Romanian a third of the test set were released as a development set instead. For Turkish additional 500 sentence development set was released. The sentences were selected from dozens of news websites and translated by professional translators. The training data consists of parallel corpora to train translation models, monolingual corpora to train language models and development sets for tuning. Some training corpora were identical from WMT 2015 (Europarl, United Nations, French-English 10⁹ corpus, Common Crawl, Russian-English parallel data provided by Yandex, Wikipedia Headlines provided by CMU) and some were update (CzEng v1.6pre, News Commentary v11, monolingual news data). Additionally, the following new corpora were added: Romanian Europarl, SETIMES2 from OPUS for Romanian-English and Turkish-English, Monolingual data sets from CommonCrawl. Source: https://paperswithcode.com/paper/findings-of-the-2016-conference-on-machine/ Image Source: https://www.aclweb.org/anthology/W16-2301.pdf