arXiv:2605.30202v2 cs.CL

A Dual-Path Architecture for Scaling Compute and Capacity in LLMs

Markus Frey, Behzad Shomali, Joachim Koehler, Mehdi Ali

제출 2026-11-05

원문 초록

Looped transformers apply a shared block multiple times and have emerged as a parameter-efficient route to scaling compute in language models. However, at fixed FLOPs a looped model has strictly less capacity than a baseline transformer. We argue that compute, the number of sequential operations applied to a hidden state, and capacity, the parameters available at a single step, are distinct scaling axes that standard transformer layers conflate. We propose a dual-path block that exposes both axes as parallel pathways within a single layer: a deep sublayer re-applied $K$ times with shared parameters, and a wide sublayer with an enlarged feed-forward network applied once, combined by independent per-token gates. At two FLOP budgets, the best dual-path configuration surpasses both iso-FLOP single-axis controls on aggregate language modelling, commonsense, and math metrics, while using 17-33% fewer parameters than the width-scaled baseline. The learned gates are directly interpretable and show systematic per-token allocation: function words and lexical content trend wide, while punctuation, symbols, and arithmetic tokens trend deep. Our results suggest that compute and capacity can be scaled separately, with the model balancing the two token by token.

제출 이력

확인된 버전만 표시합니다. 전체 제출 이력은 arXiv 원문에서 확인하세요.

  1. arXiv:2605.30202v2제출 시각 미확인

함께 살펴볼 논문

공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.

공통 태그로 연결된 공개 논문이 아직 없습니다.

토론

이 논문과 연결된 포럼 글이 아직 없습니다.