IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text
원문 초록
We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with video and text, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP). The proposed approach allows IMU2CLIP to translate human motions (as measured by IMU sensors) into their corresponding textual descriptions and videos -- while preserving the transitivity across these modalities. We explore several new IMU-based applications that IMU2CLIP enables, such as motion-based media retrieval and natural language reasoning tasks with motion data. In addition, we show that IMU2CLIP can significantly improve the downstream performance when fine-tuned for each application (e.g. activity recognition), demonstrating the universal usage of IMU2CLIP as a new pre-trained resource. Our code will be made publicly available.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
RadiomicsRetrieval: A Customizable Framework for Medical Image Retrieval Using Radiomics Features
Refining music sample identification with a self-supervised graph neural network
Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models
HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals
A Simple Contrastive Framework Of Item Tokenization For Generative Recommendation
Hierarchical Multi-Positive Contrastive Learning for Patent Image Retrieval
토론
이 논문과 연결된 포럼 글이 아직 없습니다.