Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning
원문 초록
In this paper, we present a novel approach, Momentum$^2$ Teacher, for student-teacher based self-supervised learning. The approach performs momentum update on both network weights and batch normalization (BN) statistics. The teacher's weight is a momentum update of the student, and the teacher's BN statistics is a momentum update of those in history. The Momentum$^2$ Teacher is simple and efficient. It can achieve the state of the art results (74.5\%) under ImageNet linear evaluation protocol using small-batch size(\eg, 128), without requiring large-batch training on special hardware like TPU or inefficient across GPU operation (\eg, shuffling BN, synced BN). Our implementation and pre-trained models will be given on GitHub\footnote{https://github.com/zengarden/momentum2-teacher}.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Efficient and Generalizable Speaker Diarization via Structured Pruning of Self-Supervised Models
Advancing Video Self-Supervised Learning via Image Foundation Models
Contrastive Self-Supervised Learning As Neural Manifold Packing
SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapes
Fine-tune Before Structured Pruning: Towards Compact and Accurate Self-Supervised Models for Speaker Diarization
APE: Selective Fine-tuning with Acceptance Criteria for Language Model Adaptation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.