Global Convergence of SGD On Two Layer Neural Nets
원문 초록
In this note, we consider appropriately regularized $\ell_2-$empirical risk of depth $2$ nets with any number of gates and show bounds on how the empirical loss evolves for SGD iterates on it -- for arbitrary data and if the activation is adequately smooth and bounded like sigmoid and tanh. This in turn leads to a proof of global convergence of SGD for a special class of initializations. We also prove an exponentially fast convergence rate for continuous time SGD that also applies to smooth unbounded activations like SoftPlus. Our key idea is to show the existence of Frobenius norm regularized loss functions on constant-sized neural nets which are "Villani functions" and thus be able to build on recent progress with analyzing SGD on such objectives. Most critically the amount of regularization required for our analysis is independent of the size of the net.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms
Leveraging the two timescale regime to demonstrate convergence of neural networks
Training a Two Layer ReLU Network Analytically
MiniRBT: A Two-stage Distilled Small Chinese Pre-trained Model
FinalMLP: An Enhanced Two-Stream MLP Model for CTR Prediction
Using Simple Incentives to Improve Two-Sided Fairness in Ridesharing Systems
토론
이 논문과 연결된 포럼 글이 아직 없습니다.