Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers
원문 초록
Sparsely activated transformers, such as Mixture of Experts (MoE), have received great interest due to their outrageous scaling capability which enables dramatical increases in model size without significant increases in computational cost. To achieve this, MoE models replace the feedforward sub-layer with Mixture-of-Experts sub-layer in transformers and use a gating network to route each token to its assigned experts. Since the common practice for efficient training of such models requires distributing experts and tokens across different machines, this routing strategy often incurs huge cross-machine communication cost because tokens and their assigned experts likely reside in different machines. In this paper, we propose \emph{Gating Dropout}, which allows tokens to ignore the gating network and stay at their local machines, thus reducing the cross-machine communication. Similar to traditional dropout, we also show that Gating Dropout has a regularization effect during training, resulting in improved generalization performance. We validate the effectiveness of Gating Dropout on multilingual machine translation tasks. Our results demonstrate that Gating Dropout improves a state-of-the-art MoE model with faster wall-clock time convergence rates and better BLEU scores for a variety of model sizes and datasets.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation
FuxiMT: Sparsifying Large Language Models for Chinese-Centric Multilingual Machine Translation
MoFE-Time: Mixture of Frequency Domain Experts for Time-Series Forecasting Models
A Survey on Prompt Tuning
Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate
Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Experts
토론
이 논문과 연결된 포럼 글이 아직 없습니다.