Extending Label Smoothing Regularization with Self-Knowledge Distillation
원문 초록
Inspired by the strong correlation between the Label Smoothing Regularization(LSR) and Knowledge distillation(KD), we propose an algorithm LsrKD for training boost by extending the LSR method to the KD regime and applying a softer temperature. Then we improve the LsrKD by a Teacher Correction(TC) method, which manually sets a constant larger proportion for the right class in the uniform distribution teacher. To further improve the performance of LsrKD, we develop a self-distillation method named Memory-replay Knowledge Distillation (MrKD) that provides a knowledgeable teacher to replace the uniform distribution one in LsrKD. The MrKD method penalizes the KD loss between the current model's output distributions and its copies' on the training trajectory. By preventing the model learning so far from its historical output distribution space, MrKD can stabilize the learning and find a more robust minimum. Our experiments show that LsrKD can improve LSR performance consistently at no cost, especially on several deep neural networks where LSR is ineffectual. Also, MrKD can significantly improve single model training. The experiment results confirm that the TC can help LsrKD and MrKD to boost training, especially on the networks they are failed. Overall, LsrKD, MrKD, and their TC variants are comparable to or outperform the LSR method, suggesting the broad applicability of these KD methods.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Distilled Gradual Pruning with Pruned Fine-tuning
Eyelid’s Intrinsic Motion-aware Feature Learning for Real-time Eyeblink Detection in the Wild
모델 크기 보간에서의 층 패칭 이해
Understanding Layer Patching in Model Size Interpolation
LISA: 효율적 긴 컨텍스트 추론을 위한 Linear-Indexed Sparse Attention
LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
사고 모델을 위한 온-정책 자기 지식 증류의 재고찰
Rethinking On-Policy Self-Distillation for Thinking Models
SmolWAM: 엣지 디바이스 로봇 조작을 위한 잠재 월드-액션 모델
SmolWAM: A Latent World-Action Model for Robotic Manipulation on Edge Devices
토론
이 논문과 연결된 포럼 글이 아직 없습니다.