On the High Symmetry of Neural Network Functions
원문 초록
Training neural networks means solving a high-dimensional optimization problem. Normally the goal is to minimize a loss function that depends on what is called the network function, or in other words the function that gives the network output given a certain input. This function depends on a large number of parameters, also known as weights, that depends on the network architecture. In general the goal of this optimization problem is to find the global minimum of the network function. In this paper it is discussed how due to how neural networks are designed, the neural network function present a very large symmetry in the parameter space. This work shows how the neural network function has a number of equivalent minima, in other words minima that give the same value for the loss function and the same exact output, that grows factorially with the number of neurons in each layer for feed forward neural network or with the number of filters in a convolutional neural networks. When the number of neurons and layers is large, the number of equivalent minima grows extremely fast. This will have of course consequences for the study of how neural networks converges to minima during training. This results is known, but in this paper for the first time a proper mathematical discussion is presented and an estimate of the number of equivalent minima is derived.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Scaling Laws of Dynamic High-Capacity Ride-Sharing
CUTS+: High-dimensional Causal Discovery from Irregular Time-series
CosmoPower-JAX: high-dimensional Bayesian inference with differentiable cosmological emulators
HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion
MIReAD: Simple Method for Learning High-quality Representations from Scientific Documents
High-Fidelity 3D Face Generation from Natural Language Descriptions
토론
이 논문과 연결된 포럼 글이 아직 없습니다.