EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference
원문 초록
The Mixture-of-Experts (MoE) model has emerged as a prominent architecture in the field of Large Language Models (LLMs), providing a better balance between model performance and computational efficiency. However the General Matrix Multiply (GEMM) operations and large parameters introduce challenges related to computational efficiency and communication overhead, which become throughput bottlenecks during inference. Applying a single parallelism strategy like EP, DP, TP or a straightforward combination of them to MoE usually achieves sub-optimal inference throughput. This paper introduces EPS-MoE, a novel expert pipeline scheduler for MoE that surpasses the existing parallelism schemes. Our approach optimizes the computation of MoE FeedForward Network (FFN) modules by dynamically selecting the best kernel implementation of GroupGemm and DenseGemm for different loads and adaptively overlapping these computations with communication, leading to a substantial increase in throughput. Our experimental results demonstrate at most 52.4\% improvement in prefill throughput compared to existing parallel inference methods. Specifically, our method accelerated the highly optimized DeepSeekV2 model from a claimed 100K tokens per second to at least 120K tokens per second.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
A Survey on Prompt Tuning
Mixture of Experts in Large Language Models
KAT-V1: Kwai-AutoThink Technical Report
LLM-guided Chemical Process Optimization with a Multi-Agent Approach
OmniReflect: Discovering Transferable Constitutions for LLM agents via Neuro-Symbolic Reflections
Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs
토론
이 논문과 연결된 포럼 글이 아직 없습니다.