Deep Fusion: Capturing Dependencies in Contrastive Learning via Transformer Projection Heads
원문 초록
Contrastive Learning (CL) has emerged as a powerful method for training feature extraction models using unlabeled data. Recent studies suggest that incorporating a linear projection head post-backbone significantly enhances model performance. In this work, we investigate the use of a transformer model as a projection head within the CL framework, aiming to exploit the transformer's capacity for capturing long-range dependencies across embeddings to further improve performance. Our key contributions are fourfold: First, we introduce a novel application of transformers in the projection head role for contrastive learning, marking the first endeavor of its kind. Second, our experiments reveal a compelling "Deep Fusion" phenomenon where the attention mechanism progressively captures the correct relational dependencies among samples from the same class in deeper layers. Third, we provide a theoretical framework that explains and supports this "Deep Fusion" behavior. Finally, we demonstrate through experimental results that our model achieves superior performance compared to the existing approach of using a feed-forward layer.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Revisiting Cross-Modal Knowledge Distillation: A Disentanglement Approach for RGBD Semantic Segmentation
Enhancing Contrastive Learning-based Electrocardiogram Pretrained Model with Patient Memory Queue
Overview of the TalentCLEF 2025: Skill and Job Title Intelligence for Human Capital Management
Similarity-Guided Diffusion for Contrastive Sequential Recommendation
IPCL: Iterative Pseudo-Supervised Contrastive Learning to Improve Self-Supervised Feature Representation
DOMFN: A Divergence-Orientated Multi-Modal Fusion Network for Resume Assessment
토론
이 논문과 연결된 포럼 글이 아직 없습니다.