Multimodal Machine Translation

Multi30K is a large-scale multilingual multimodal dataset for interdisciplinary machine learning research. It extends the Flickr30K dataset with German translations created by professional translators over a subset of the English descriptions, and descriptions crowdsourced independently of the original English descriptions. The dataset was introduced to stimulate multilingual multimodal research.

Hindi Visual Genome is a multimodal dataset consisting of text and images suitable for English-Hindi multimodal machine translation task and multimodal research. Source: Hindi Visual Genome: A Dataset for Multimodal English-to-Hindi Machine Translation Image Source: https://ufal.mff.cuni.cz/hindi-visual-genome

MultiSubs is a dataset of multilingual subtitles gathered from the OPUS OpenSubtitles dataset, which in turn was sourced from opensubtitles.org. We have supplemented some text fragments (visually salient nouns in this release) within the subtitles with web images, where the word sense of the fragment has been disambiguated using a cross-lingual approach. We have introduced a fill-in-the-blank task and a lexical translation task to demonstrate the utility of the dataset. Please refer to our paper for a more detailed description of the dataset and tasks. Multisubs will benefit research on visual grounding of words especially in the context of free-form sentence. Josiah Wang, Pranava Madhyastha, Josiel Figueiredo, Chiraag Lala, Lucia Specia (2021). MultiSubs: A Large-scale Multimodal and Multilingual Dataset. CoRR, abs/2103.01910. Available at: [https://arxiv.org/abs/2103.01910] (https://arxiv.org/abs/2103.01910)

A dataset that contains the description of an image or a section within the image in Hausa and its equivalent in English. Hausa, a Chadic language, is a member of the Afro-Asiatic language family. It is estimated that about 100 to 150 million people speak the language, with more than 80 million indigenous speakers. The dataset comprises 32,923 images and their descriptions that are divided into training, development, test, and challenge test set. The Hausa Visual Genome is the first dataset of its kind and can be used for Hausa-English machine translation, multi-modal research, and image description, among various other natural language processing and generation tasks.

The Biomedical Translation Shared Task was first introduced at the First Conference of Machine Translation. The task aims to evaluate systems for the translation of biomedical titles and abstracts from scientific publications. The data includes three language pairs (English ↔ Portuguese, English ↔ Spanish, English ↔ French) and two sub-domains of biological sciences and health sciences. The training data consists mainly of the Scielo corpus, a parallel collection of scientific publications composed of either titles, abstracts or title and abstracts which were retrieved from the Scielo database. For the Scielo corpus, a parallel documents are provided for all language pairs in the two sub-domains, except for the English ↔ French, where only health was considered, as there were inadequate parallel documents available for biology in that pair. The training data was aligned using the GMA alignment tool. Additionally, a corpus of parallel titles from MEDLINEⓇ for all three language pairs were provided as well as monolingual documents for the four languages, retrieved from the Scielo database. These consist of documents in the Scielo database which have no corresponding document in another language. The test set consisted of 500 documents (title and abstract) for each of the two directions of each language pair. None of the test documents was included in the training data and there is no overlap of documents between the test sets for any language pair, translation direction and sub-domain. Source: http://www.statmt.org/wmt16/index.html Image Source: https://www.aclweb.org/anthology/W16-2301.pdf

The IT Translation Task is a shared task introduced in the First Conference on Machine Translation. Compared to WMT 2016 News, this task brought several novelties to WMT: * 4 out of the 7 langauges of the IT task are new in WMT, * adaptation to the IT domain with its specifics such as frequent named entities (mostly menu items, names of products and companies) and technical jargon, * adaptation to translation of answers in helpdesk service setting (many of the sentences are instructions with imperative verbs, which is very rare in the News translation task). The test set consisted of 1000 answers from the Batch 3 of the QTLeap Corpus. The in-domain training data contained 2000 answers from the Batches 1 and 2 and also localization files from several open-source projects (LibreOffice, KDE, VLC) and bilingual dictionaries of IT-related terms extracted from Wikipedia. The out-of-domain training data contained all the corpora from the WMT 2016 News, plus PaCo2-EuEn Basque-English corpus and SETimes with Bulgarian-English parallel sentences. “Constrained” systems were restricted to use only these training data provided by the organizers. The task was evaluated on the following language pairs: * English → Bulgarian * English → Czech * English → German * English → Spanish * English → Basque * English → Dutch * English → Portuguese Source: http://www.statmt.org/wmt16/index.html Image Source: Bojar et al