arXiv:2501.02631

Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for Estonian

Aleksei Dorkin, Taido Purason, Kairit Sirts

제출 2025-01-05

원문 초록

Adapting multilingual language models to specific languages can enhance both their efficiency and performance. In this study, we explore how modifying the vocabulary of a multilingual encoder model to better suit the Estonian language affects its downstream performance on the Named Entity Recognition (NER) task. The motivations for adjusting the vocabulary are twofold: practical benefits affecting the computational cost, such as reducing the input sequence length and the model size, and performance enhancements by tailoring the vocabulary to the particular language. We evaluate the effectiveness of two vocabulary adaptation approaches -- retraining the tokenizer and pruning unused tokens -- and assess their impact on the model's performance, particularly after continual training. While retraining the tokenizer degraded the performance of the NER task, suggesting that longer embedding tuning might be needed, we observed no negative effects on pruning.

함께 살펴볼 논문

공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.

arXiv:2504.16007공통 태그 Named Entity Recognition (NER), NER, named-entity-recognition, Named Entity Recognition공식 코드 · Papers with Code

Methods for Recognizing Nested Terms

토론

이 논문과 연결된 포럼 글이 아직 없습니다.