Sharp asymptotics on the compression of two-layer neural networks
원문 초록
In this paper, we study the compression of a target two-layer neural network with N nodes into a compressed network with M<N nodes. More precisely, we consider the setting in which the weights of the target network are i.i.d. sub-Gaussian, and we minimize the population L_2 loss between the outputs of the target and of the compressed network, under the assumption of Gaussian inputs. By using tools from high-dimensional probability, we show that this non-convex problem can be simplified when the target network is sufficiently over-parameterized, and provide the error rate of this approximation as a function of the input dimension and N. In this mean-field limit, the simplified objective, as well as the optimal weights of the compressed network, does not depend on the realization of the target network, but only on expected scaling factors. Furthermore, for networks with ReLU activation, we conjecture that the optimum of the simplified optimization problem is achieved by taking weights on the Equiangular Tight Frame (ETF), while the scaling of the weights and the orientation of the ETF depend on the parameters of the target network. Numerical evidence is provided to support this conjecture.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms
Leveraging the two timescale regime to demonstrate convergence of neural networks
Training a Two Layer ReLU Network Analytically
MiniRBT: A Two-stage Distilled Small Chinese Pre-trained Model
FinalMLP: An Enhanced Two-Stream MLP Model for CTR Prediction
Using Simple Incentives to Improve Two-Sided Fairness in Ridesharing Systems
토론
이 논문과 연결된 포럼 글이 아직 없습니다.