Investigating Language-Specific Calibration For Pruning Multilingual Large Language Models
원문 초록
Recent advances in large language model (LLM) pruning have shown state-of-the-art (SotA) compression results in post-training and retraining-free settings while maintaining high predictive performance. However, previous research mainly considered calibrating based on English text, despite the multilingual nature of modern LLMs and their frequent use in non-English languages. In this paper, we set out to investigate calibrating the pruning of multilingual language models for monolingual applications. We present the first comprehensive empirical study, comparing different calibration languages for pruning multilingual models across diverse languages, tasks, models, and SotA pruning techniques. Our results offer practical suggestions, for example, calibrating in the target language can efficiently retain the language modeling capability but does not necessarily benefit downstream tasks. Through further analysis of latent subspaces, pruning masks, and individual neurons within pruned models, we find that while pruning generally preserves strong language-specific features, it may fail to retain language-specific neuron activation patterns and subtle, language-agnostic features associated with knowledge and reasoning that are needed for complex tasks.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks
Assay2Mol: large language model-based drug design using BioAssay context
Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery
Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models
mTSBench: Benchmarking Multivariate Time Series Anomaly Detection and Model Selection at Scale
AgentStealth: Reinforcing Large Language Model for Anonymizing User-generated Text
토론
이 논문과 연결된 포럼 글이 아직 없습니다.