Annihilation of Spurious Minima in Two-Layer ReLU Networks
원문 초록
We study the optimization problem associated with fitting two-layer ReLU neural networks with respect to the squared loss, where labels are generated by a target network. Use is made of the rich symmetry structure to develop a novel set of tools for studying the mechanism by which over-parameterization annihilates spurious minima. Sharp analytic estimates are obtained for the loss and the Hessian spectrum at different minima, and it is proved that adding neurons can turn symmetric spurious minima into saddles; minima of lesser symmetry require more neurons. Using Cauchy's interlacing theorem, we prove the existence of descent directions in certain subspaces arising from the symmetry structure of the loss function. This analytic approach uses techniques, new to the field, from algebraic geometry, representation theory and symmetry breaking, and confirms rigorously the effectiveness of over-parameterization in making the associated loss landscape accessible to gradient-based methods. For a fixed number of neurons and inputs, the spectral results remain true under symmetry breaking perturbation of the target.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms
Leveraging the two timescale regime to demonstrate convergence of neural networks
Training a Two Layer ReLU Network Analytically
MiniRBT: A Two-stage Distilled Small Chinese Pre-trained Model
FinalMLP: An Enhanced Two-Stream MLP Model for CTR Prediction
Using Simple Incentives to Improve Two-Sided Fairness in Ridesharing Systems
토론
이 논문과 연결된 포럼 글이 아직 없습니다.