Normalization of Input-output Shared Embeddings in Text Generation Models
원문 초록
Neural Network based models have been state-of-the-art models for various Natural Language Processing tasks, however, the input and output dimension problem in the networks has still not been fully resolved, especially in text generation tasks (e.g. Machine Translation, Text Summarization), in which input and output both have huge sizes of vocabularies. Therefore, input-output embedding weight sharing has been introduced and adopted widely, which remains to be improved. Based on linear algebra and statistical theories, this paper locates the shortcoming of existed input-output embedding weight sharing method, then raises methods for improving input-output weight shared embedding, among which methods of normalization of embedding weight matrices show best performance. These methods are nearly computational cost-free, can get combined with other embedding techniques, and show good effectiveness when applied on state-of-the-art Neural Network models. For Transformer-big models, the normalization techniques can get at best 0.6 BLEU improvement compared to the original version of model on WMT'16 En-De dataset, and similar BLEU improvements on IWSLT 14' datasets. For DynamicConv models, 0.5 BLEU improvement can be attained on WMT'16 En-De dataset, and 0.41 BLEU improvement on IWSLT 14' De-En translation task is achieved.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Prompting LLMs: Length Control for Isometric Machine Translation
CycleDistill: Bootstrapping Machine Translation using LLMs with Cyclical Distillation
Gender Bias in English-to-Greek Machine Translation
TACTIC: Translation Agents with Cognitive-Theoretic Interactive Collaboration
Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations
Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.