Crowdsourced Phrase-Based Tokenization for Low-Resourced Neural Machine Translation: The Case of Fon Language
원문 초록
Building effective neural machine translation (NMT) models for very low-resourced and morphologically rich African indigenous languages is an open challenge. Besides the issue of finding available resources for them, a lot of work is put into preprocessing and tokenization. Recent studies have shown that standard tokenization methods do not always adequately deal with the grammatical, diacritical, and tonal properties of some African languages. That, coupled with the extremely low availability of training samples, hinders the production of reliable NMT models. In this paper, using Fon language as a case study, we revisit standard tokenization methods and introduce Word-Expressions-Based (WEB) tokenization, a human-involved super-words tokenization strategy to create a better representative vocabulary for training. Furthermore, we compare our tokenization strategy to others on the Fon-French and French-Fon translation tasks.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Do Not Change Me: On Transferring Entities Without Modification in Neural Machine Translation -- a Multilingual Perspective
Calibrating Translation Decoding with Quality Estimation on LLMs
Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation
ConECT Dataset: Overcoming Data Scarcity in Context-Aware E-Commerce MT
In-Domain African Languages Translation Using LLMs and Multi-armed Bandits
THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.