JPmHC Dynamical Isometry via Orthogonal Hyper-Connections

원문 초록

The hyper-connection paradigm separates the skip-connections of transformers into $n$ parallel streams on which a linear mixer acts. Constraining the mixer to bistochastic matrices has proven effective in preventing gradient explosion but risk collapsing onto low-rank configurations, limiting expressivity. We therefore propose JPmHC (Jacobian-spectrum Preserving manifold-constrained Hyper-Connections). Our framework triages manifold constraints by computing the Jacobian singular spectrum of the end-to-end network in the mean-field limit and their spectral stability. It consists of: (i) an operator-valued free-probability library that predicts the Jacobian singular spectrum; (ii) an orthogonal mixer computed by a fixed-point Cayley iteration, which enables dynamical isometry; (iii) implicit differentiation of the Sinkhorn projection of the bistochastic baseline, with $O(1)$ backward operations; and (iv) a sensitivity theory giving the training budget before a bistochastic mixer collapses. The library is written in Lean, which certifies the mathematics and the algorithms. We validate JPmHC with TRM on ARC-AGI and nano-GPT on FineWeb-Edu: the theory predicts spectral collapse for TRM and stability for nano-GPT as observed in our experiments.