Embedding Principle of Loss Landscape of Deep Neural Networks
원문 초록
Understanding the structure of loss landscape of deep neural networks (DNNs)is obviously important. In this work, we prove an embedding principle that the loss landscape of a DNN "contains" all the critical points of all the narrower DNNs. More precisely, we propose a critical embedding such that any critical point, e.g., local or global minima, of a narrower DNN can be embedded to a critical point/hyperplane of the target DNN with higher degeneracy and preserving the DNN output function. The embedding structure of critical points is independent of loss function and training data, showing a stark difference from other nonconvex problems such as protein-folding. Empirically, we find that a wide DNN is often attracted by highly-degenerate critical points that are embedded from narrow DNNs. The embedding principle provides an explanation for the general easy optimization of wide DNNs and unravels a potential implicit low-complexity regularization during the training. Overall, our work provides a skeleton for the study of loss landscape of DNNs and its implication, by which a more exact and comprehensive understanding can be anticipated in the near
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MegaFold: System-Level Optimizations for Accelerating Protein Structure Prediction Models
Probably Approximately Correct Labels
ReQFlow: Rectified Quaternion Flow for Efficient and High-Quality Protein Backbone Generation
PyMOLfold: Interactive Protein and Ligand Structure Prediction in PyMOL
Mask prior-guided denoising diffusion improves inverse protein folding
Bridge-IF: Learning Inverse Protein Folding with Markov Bridges
토론
이 논문과 연결된 포럼 글이 아직 없습니다.