Experiments with Rich Regime Training for Deep Learning
원문 초록
In spite of advances in understanding lazy training, recent work attributes the practical success of deep learning to the rich regime with complex inductive bias. In this paper, we study rich regime training empirically with benchmark datasets, and find that while most parameters are lazy, there is always a small number of active parameters which change quite a bit during training. We show that re-initializing (resetting to their initial random values) the active parameters leads to worse generalization. Further, we show that most of the active parameters are in the bottom layers, close to the input, especially as the networks become wider. Based on such observations, we study static Layer-Wise Sparse (LWS) SGD, which only updates some subsets of layers. We find that only updating the top and bottom layers have good generalization and, as expected, only updating the top layers yields a fast algorithm. Inspired by this, we investigate probabilistic LWS-SGD, which mostly updates the top layers and occasionally updates the full network. We show that probabilistic LWS-SGD matches the generalization performance of vanilla SGD and the back-propagation time can be 2-5 times more efficient.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Experts
AGTCNet: A Graph-Temporal Approach for Principled Motor Imagery EEG Classification
Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation
Parameter-free approximate equivariance for tasks with finite group symmetry
Positional Encoding meets Persistent Homology on Graphs
Information Locality as an Inductive Bias for Neural Language Models
토론
이 논문과 연결된 포럼 글이 아직 없습니다.