Papers with Code 아카이브

L2 Regularization

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크: 0
평가표: 0
보존된 평가 기록: 0

$L_{2}$ Regularization or Weight Decay, is a regularization technique applied to the weights of a neural network. We minimize a loss function compromising both the primary loss function and a penalty on the $L\_{2}$ Norm of the weights:

$$L\_{new}\left(w\right) = L\_{original}\left(w\right) + \lambda{w^{T}w}$$

where $\lambda$ is a value determining the strength of the penalty (encouraging smaller weights).

Weight decay can be incorporated directly into the weight update rule, rather than just implicitly by defining it through to objective function. Often weight decay refers to the implementation where we specify it directly in the weight update rule (whereas L2 regularization is usually the implementation which is specified in the objective function).

기법

관련 논문에서 소개한 기법입니다. 이 태스크에서의 평가 여부는 평가 결과에서 확인하세요.

Activation Regularization

Activation Regularization (AR), or $L\_{2}$ activation regularization, is regularization performed on activations as opposed to weights. It is usually used in conjunction with RNNs. It is defined as: $$\alpha{L}\_{2}\left(m\circ{h\_{t}}\right) $$ where $m$ is a dropout mask used by later parts of the model, $L\_{2}$ is the $L\_{2}$ norm, and $h_{t}$ is the output of an RNN at timestep $t$, and $\alpha$ is a scaling coefficient. When applied to the output of a dense layer, AR penalizes activations that are substantially away from 0, encouraging activations to remain small.

Revisiting Activation Regularization for Language RNNs

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.