Minibatching Offers Improved Generalization Performance for Second Order Optimizers
원문 초록
Training deep neural networks (DNNs) used in modern machine learning is computationally expensive. Machine learning scientists, therefore, rely on stochastic first-order methods for training, coupled with significant hand-tuning, to obtain good performance. To better understand performance variability of different stochastic algorithms, including second-order methods, we conduct an empirical study that treats performance as a response variable across multiple training sessions of the same model. Using 2-factor Analysis of Variance (ANOVA) with interactions, we show that batch size used during training has a statistically significant effect on the peak accuracy of the methods, and that full batch largely performed the worst. In addition, we found that second-order optimizers (SOOs) generally exhibited significantly lower variance at specific batch sizes, suggesting they may require less hyperparameter tuning, leading to a reduced overall time to solution for model training.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation
Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis
SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Online Covariance Matrix Estimation in Sketched Newton Methods
Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization
Alternating Iteratively Reweighted $\ell_1$ and Subspace Newton Algorithms for Nonconvex Sparse Optimization
토론
이 논문과 연결된 포럼 글이 아직 없습니다.