Adaptive Preconditioners Trigger Loss Spikes in Adam
원문 초록
Loss spikes emerge commonly during training across neural networks of varying architectures and scales when using the Adam optimizer. In this work, we investigate the underlying mechanism responsible for Adam spikes. While previous explanations attribute these phenomena to the lower-loss-as-sharper characteristics of the loss landscape, our analysis reveals that Adam's adaptive preconditioners themselves can trigger spikes. Specifically, we identify a critical regime where squared gradients become substantially smaller than the second-order moment estimates, causing the latter to undergo a $\beta_2$-exponential decay and to respond sluggishly to current gradient information. This mechanism can push the maximum eigenvalue of the preconditioned Hessian beyond the classical stability threshold $2/\eta$ for a sustained period, inducing instability. This instability further leads to an alignment between the gradient and the maximum eigendirection, and a loss spike occurs precisely when the gradient-directional curvature exceeds $2/\eta$. We verify this mechanism through extensive experiments on fully connected networks, convolutional networks, and Transformer architectures.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Attributes Shape the Embedding Space of Face Recognition Models
Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models
Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations
MambaFusion: Height-Fidelity Dense Global Fusion for Multi-modal 3D Object Detection
The Trilemma of Truth in Large Language Models
Style-Aligned Image Composition for Robust Detection of Abnormal Cells in Cytopathology
토론
이 논문과 연결된 포럼 글이 아직 없습니다.