Lifting the Curse of Multilinguality by Pre-training Modular Transformers
원문 초록
Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this issue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. In contrast with prior work that learns language-specific components post-hoc, we pre-train the modules of our Cross-lingual Modular (X-Mod) models from the start. Our experiments on natural language inference, named entity recognition and question answering show that our approach not only mitigates the negative interference between languages, but also enables positive transfer, resulting in improved monolingual and cross-lingual performance. Furthermore, our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
A Neural Named Entity Recognition and Multi-Type Normalization Tool for Biomedical Text Mining
Selecting and Merging: Towards Adaptable and Scalable Named Entity Recognition with Large Language Models
Label-Guided In-Context Learning for Named Entity Recognition
Named Entity Recognition in Historical Italian: The Case of Giacomo Leopardi's Zibaldone
Flippi: End To End GenAI Assistant for E-Commerce
Better Semi-supervised Learning for Multi-domain ASR Through Incremental Retraining and Data Filtering
토론
이 논문과 연결된 포럼 글이 아직 없습니다.