arXiv:1706.05350
L2 Regularization versus Batch and Weight Normalization
원문 초록
Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no regularizing effect when combined with normalization. Instead, regularization has an influence on the scale of weights, and thereby on the effective learning rate. We investigate this dependence, both in theory, and experimentally. We show that popular optimization methods such as ADAM only partially eliminate the influence of normalization on the learning rate. This leads to a discussion on other ways to mitigate this issue.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
DACN: Dual-Attention Convolutional Network for Hyperspectral Image Super-Resolution
WALINET: A water and lipid identification convolutional Neural Network for nuisance signal removal in 1H MR Spectroscopic Imaging
Monkeypox disease recognition model based on improved SE-InceptionV3
Convergence of a L2 regularized Policy Gradient Algorithm for the Multi Armed Bandit
Prevalidated ridge regression is a highly-efficient drop-in replacement for logistic regression for high-dimensional data
Gradient-based bilevel optimization for multi-penalty Ridge regression through matrix differential calculus
토론
이 논문과 연결된 포럼 글이 아직 없습니다.