AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations
원문 초록
Self-supervision has shown great potential for audio-visual speech recognition by vastly reducing the amount of labeled data required to build good systems. However, existing methods are either not entirely end-to-end or do not train joint representations of both modalities. In this paper, we introduce AV-data2vec which addresses these challenges and builds audio-visual representations based on predicting contextualized representations which has been successful in the uni-modal case. The model uses a shared transformer encoder for both audio and video and can combine both modalities to improve speech recognition. Results on LRS3 show that AV-data2vec consistently outperforms existing methods under all settings with the same amount of data and model size.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition
Cocktail-Party Audio-Visual Speech Recognition
The Multimodal Information Based Speech Processing (MISP) 2025 Challenge: Audio-Visual Diarization and Recognition
Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach
Part-based Lipreading for Audio-Visual Speech Recognition
GigaAM: Efficient Self-Supervised Learner for Speech Recognition
토론
이 논문과 연결된 포럼 글이 아직 없습니다.