VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
원문 초록
Although speech is a simple and effective way for humans to communicate with the outside world, a more realistic speech interaction contains multimodal information, e.g., vision, text. How to design a unified framework to integrate different modal information and leverage different resources (e.g., visual-audio pairs, audio-text pairs, unlabeled speech, and unlabeled text) to facilitate speech representation learning was not well explored. In this paper, we propose a unified cross-modal representation learning framework VATLM (Visual-Audio-Text Language Model). The proposed VATLM employs a unified backbone network to model the modality-independent information and utilizes three simple modality-dependent modules to preprocess visual, speech, and text inputs. In order to integrate these three modalities into one shared semantic space, VATLM is optimized with a masked prediction task of unified tokens, given by our proposed unified tokenizer. We evaluate the pre-trained VATLM on audio-visual related downstream tasks, including audio-visual speech recognition (AVSR), visual speech recognition (VSR) tasks. Results show that the proposed VATLM outperforms previous the state-of-the-art models, such as audio-visual pre-trained AV-HuBERT model, and analysis also demonstrates that VATLM is capable of aligning different modalities into the same space. To facilitate future research, we release the code and pre-trained models at https://aka.ms/vatlm.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
SEED: A Structural Encoder for Embedding-Driven Decoding in Time Series Prediction with LLMs
Multimodal Representation Learning and Fusion
LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization
Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition
GeoRecon: Graph-Level Representation Learning for 3D Molecules via Reconstruction-Based Pretraining
토론
이 논문과 연결된 포럼 글이 아직 없습니다.