Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models
원문 초록
Self-Supervised Learning (SSL) is at the core of training modern large machine learning models, providing a scheme for learning powerful representations that can be used in a variety of downstream tasks. However, SSL strategies must be adapted to the type of training data and downstream tasks required. We propose RS3L ("Re-simulation-based self-supervised representation learning"), a novel simulation-based SSL strategy that employs a method of re-simulation to drive data augmentation for contrastive learning in the physical sciences, particularly, in fields that rely on stochastic simulators. By intervening in the middle of the simulation process and re-running simulation components downstream of the intervention, we generate multiple realizations of an event, thus producing a set of augmentations covering all physics-driven variations available in the simulator. Using experiments from high-energy physics, we explore how this strategy may enable the development of a foundation model; we show how RS3L pre-training enables powerful performance in downstream tasks such as discrimination of a variety of objects and uncertainty mitigation. In addition to our results, we make the RS3L dataset publicly available for further studies on how to improve SSL strategies.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
IPCL: Iterative Pseudo-Supervised Contrastive Learning to Improve Self-Supervised Feature Representation
Self-Supervised Learning for Multimedia Recommendation
Similarity-Guided Diffusion for Contrastive Sequential Recommendation
Exploring Non-contrastive Self-supervised Representation Learning for Image-based Profiling
Contrastive Self-Supervised Learning As Neural Manifold Packing
DOMFN: A Divergence-Orientated Multi-Modal Fusion Network for Resume Assessment
토론
이 논문과 연결된 포럼 글이 아직 없습니다.