Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition
원문 초록
Although many previous studies have carried out multimodal learning with real-time MRI data that captures the audio-visual kinematics of the vocal tract during speech, these studies have been limited by their reliance on multi-speaker corpora. This prevents such models from learning a detailed relationship between acoustics and articulation due to considerable cross-speaker variability. In this study, we develop unimodal audio and video models as well as multimodal models for phoneme recognition using a long-form single-speaker MRI corpus, with the goal of disentangling and interpreting the contributions of each modality. Audio and multimodal models show similar performance on different phonetic manner classes but diverge on places of articulation. Interpretation of the models' latent space shows similar encoding of the phonetic space across audio and multimodal models, while the models' attention weights highlight differences in acoustic and articulatory timing for certain phonemes.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
LASER: Learning by Aligning Self-supervised Representations of Speech for Improving Content-related Tasks
SCORE: Self-supervised Correspondence Fine-tuning for Improved Content Representations
Allophant: Cross-lingual Phoneme Recognition with Articulatory Attributes
FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning
Predicting within and across language phoneme recognition performance of self-supervised learning speech pre-trained models
Text-Aware End-to-end Mispronunciation Detection and Diagnosis
토론
이 논문과 연결된 포럼 글이 아직 없습니다.