Beyond Embedding Transfer: Component Roles in Grokking Transfer and Stability

원문 초록

Warm-start transfer accelerates delayed generalization (grokking) on algorithmic tasks, yet how internal blocks, residual-stream depth, group non-commutativity, and continual optimization govern transferred circuits remains unclear. Across ten scale-matched batteries ($>1{,}800$ trajectories) on abelian $\mathbb{Z}_{113}$ ($L \in \{1,2,3\}$) and non-abelian $S_5$, transferring internal blocks alongside boundary carriers lifts early accuracy by $5.46$ percentage points and cuts validation confirmation latency by $616.7 \pm 129.0$ steps in 1-layer models ($p_{\rm Holm}=0.004$). In 2-layer models, full transfer compresses trigger onset $8.1\times$ ($+704.7 \pm 84.3$ integral units), whereas omitting donor embeddings incurs a $-4{,}475.6 \pm 76.8$-unit interface deficit. On ternary $S_5$ conjugation ($aba^{-1}$), two-token architectures stall in a unary conjugacy-class shortcut ($\mathrm{Cl}(aba^{-1})=\mathrm{Cl}(b)$, $34.48\% \pm 6.19\%$ peak accuracy). Combining three-token causal routing ($[a,b,a]$) with Peter--Weyl width ($d_{\rm model}=256 > \sum_\lambda d_\lambda^2 = 120$) reaches $97.97\% \pm 1.01\%$ ($80.08\% \pm 7.81\%$ at equal $427\text{k}$ parameters) with $3.2\times$ faster onset than scratch. Under continued AdamW optimization, unshielded circuits relapse by $11.85\%\text{--}80.58\%$ drawdown, and depth-$L$ stability follows a graded $(L-1)$-prefix pinning hierarchy ($75.99\% \to 31.26\% \to 0.32\% \to 0.04\%$ drawdown at $L=3$). Finally, \textbf{Closed-Loop Curvature-Anchored Consolidation} ($\eta_k = \eta_0(1 - A_{\rm val}(k))$ with monotonic AMSGrad second moments) holds saturated drawdown to $0.00\%\text{--}0.12\%$ and lifts pre-saturation terminal accuracy by $+1.67\text{--}5.37$ points over fixed-budget micro-stepping ($87.06\% \pm 0.94\%$ vs.\ $81.69\% \pm 0.90\%$ on $S_5$, $p_{\rm Holm}=0.002$).