Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss
원문 초록
Running prompt tokens through fewer routed experts during prefill makes mixture-of-experts (MoE) serving cheaper, but what it damages has not been measured. We locate the damage with token-, layer- and attention-path interventions in four MoE families on short-answer question answering. At half the experts the F1 loss is mostly answer form: the model still finds the answer but no longer answers briefly. The loss is made at a few tokens: the two chat-template tokens before the answer in Qwen3-30B-A3B, and the first 8-32 tokens, where the instruction sits, in DeepSeek-V2-Lite. Keeping them at full experts removes most of the F1 loss and about half and a fifth of the increase in answer NLL. Reduction changes the first-token sink statistics by less than 4%, and restoring super experts or massive activations recovers at most a quarter of the loss. We then test placement on real prompts with about 4% of tokens at full experts. Putting them on the task interface keeps F1 within one point of the unmodified model, while the same budget scattered in the documents is about 3 points worse in both prespecified primary tests of this comparison. Time to first token falls about 14% on 11K-token prompts.
제출 이력
확인된 버전만 표시합니다. 전체 제출 이력은 arXiv 원문에서 확인하세요.
- arXiv:2610.22454v3제출 시각 미확인
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
공통 태그로 연결된 공개 논문이 아직 없습니다.
토론
이 논문과 연결된 포럼 글이 아직 없습니다.