Explicit Mutual Information Maximization for Self-Supervised Learning
원문 초록
Recently, self-supervised learning (SSL) has been extensively studied. Theoretically, mutual information maximization (MIM) is an optimal criterion for SSL, with a strong theoretical foundation in information theory. However, it is difficult to directly apply MIM in SSL since the data distribution is not analytically available in applications. In practice, many existing methods can be viewed as approximate implementations of the MIM criterion. This work shows that, based on the invariance property of MI, explicit MI maximization can be applied to SSL under a generic distribution assumption, i.e., a relaxed condition of the data distribution. We further illustrate this by analyzing the generalized Gaussian distribution. Based on this result, we derive a loss function based on the MIM criterion using only second-order statistics. We implement the new loss for SSL and demonstrate its effectiveness via extensive experiments.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
RetFiner: A Vision-Language Refinement Scheme for Retinal Foundation Models
Boosting Generative Adversarial Transferability with Self-supervised Vision Transformer Features
Post-training for Deepfake Speech Detection
Topology-Aware Modeling for Unsupervised Simulation-to-Reality Point Cloud Recognition
Efficient and Generalizable Speaker Diarization via Structured Pruning of Self-Supervised Models
Pushing the Performance of Synthetic Speech Detection with Kolmogorov-Arnold Networks and Self-Supervised Learning Models
토론
이 논문과 연결된 포럼 글이 아직 없습니다.