MoEC: Mixture of Expert Clusters
원문 초록
Sparsely Mixture of Experts (MoE) has received great interest due to its promising scaling capability with affordable computational overhead. MoE converts dense layers into sparse experts, and utilizes a gated routing network to make experts conditionally activated. However, as the number of experts grows, MoE with outrageous parameters suffers from overfitting and sparse data allocation. Such problems are especially severe on tasks with limited data, thus hindering the progress for MoE models to improve performance by scaling up. In this work, we propose Mixture of Expert Clusters - a general approach to enable expert layers to learn more diverse and appropriate knowledge by imposing variance-based constraints on the routing stage. We further propose a cluster-level expert dropout strategy specifically designed for the expert cluster structure. Our experiments reveal that MoEC could improve performance on machine translation and natural language understanding tasks, and raise the performance upper bound for scaling up experts under limited data. We also verify that MoEC plays a positive role in mitigating overfitting and sparse data allocation.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
A Case Study of Cross-Lingual Zero-Shot Generalization for Classical Languages in LLMs
Semantic similarity estimation for domain specific data using BERT and other techniques
THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation
FuxiMT: Sparsifying Large Language Models for Chinese-Centric Multilingual Machine Translation
Vision Language Action Models in Robotic Manipulation: A Systematic Review
MoFE-Time: Mixture of Frequency Domain Experts for Time-Series Forecasting Models
토론
이 논문과 연결된 포럼 글이 아직 없습니다.