Genus expansion for non-linear random matrix ensembles with applications to neural networks
원문 초록
We present a unified approach to studying certain non-linear random matrix ensembles and associated random neural networks at initialization. This begins with a novel series expansion for neural networks which generalizes Fa\'a di Bruno's formula to an arbitrary number of compositions. The role of monomials is played by random multilinear maps indexed by directed graphs, whose edges correspond to random matrices. Crucially, this expansion linearizes the effect of the activation functions, allowing for the direct application of Wick's principle and the genus expansion technique. As an application, we prove several results about neural networks with random weights. We first give a new proof of the fact that they converge to Gaussian processes as their width tends to infinity. Secondly, we quantify the rate of convergence of the Neural Tangent Kernel to its deterministic limit in Frobenius norm. Finally, we compute the moments of the limiting spectral distribution of the Jacobian (only the first two of which were previously known), expressing them as sums over non-crossing partitions. All of these results are then generalised to the case of neural networks with sparse and non-Gaussian weights, under moment assumptions.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Statistical Machine Learning for Astronomy -- A Textbook
LLINBO: Trustworthy LLM-in-the-Loop Bayesian Optimization
MathOptAI.jl: Embed trained machine learning predictors into JuMP models
Tailored Architectures for Time Series Forecasting: Evaluating Deep Learning Models on Gaussian Process-Generated Data
The Currents of Conflict: Decomposing Conflict Trends with Gaussian Processes
Few-Shot Speech Deepfake Detection Adaptation with Gaussian Processes
토론
이 논문과 연결된 포럼 글이 아직 없습니다.