How Transformers Get Rich: Approximation and Dynamics Analysis
원문 초록
Transformers have demonstrated exceptional in-context learning capabilities, yet the theoretical understanding of the underlying mechanisms remains limited. A recent work (Elhage et al., 2021) identified a ``rich'' in-context mechanism known as induction head, contrasting with ``lazy'' $n$-gram models that overlook long-range dependencies. In this work, we provide both approximation and dynamics analyses of how transformers implement induction heads. In the {\em approximation} analysis, we formalize both standard and generalized induction head mechanisms, and examine how transformers can efficiently implement them, with an emphasis on the distinct role of each transformer submodule. For the {\em dynamics} analysis, we study the training dynamics on a synthetic mixed target, composed of a 4-gram and an in-context 2-gram component. This controlled setting allows us to precisely characterize the entire training process and uncover an {\em abrupt transition} from lazy (4-gram) to rich (induction head) mechanisms as training progresses.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Assay2Mol: large language model-based drug design using BioAssay context
Universal pre-training by iterated random computation
Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Conquering the Retina: Bringing Visual in-Context Learning to OCT
Evaluating Cell Type Inference in Vision Language Models Under Varying Visual Context
PRO-V: An Efficient Program Generation Multi-Agent System for Automatic RTL Verification
토론
이 논문과 연결된 포럼 글이 아직 없습니다.