An Analysis of Semantically-Aligned Speech-Text Embeddings
원문 초록
Embeddings play an important role in end-to-end solutions for multi-modal language processing problems. Although there has been some effort to understand the properties of single-modality embedding spaces, particularly that of text, their cross-modal counterparts are less understood. In this work, we study some intrinsic properties of a joint speech-text embedding space, constructed by minimizing the distance between paired utterance and transcription inputs in a teacher-student model setup, that are informative for several prominent use cases. We found that incorporating automatic speech recognition through both pretraining and multitask scenarios aid semantic alignment significantly, resulting in more tightly coupled embeddings. To analyse cross-modal embeddings we utilise a quantitative retrieval accuracy metric for semantic alignment, zero-shot classification for generalisability, and probing of the encoders to observe the extent of knowledge transfer from one modality to another.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Mlphon: A Multifunctional Grapheme-Phoneme Conversion Tool Using Finite State Transducers
Transformer Based Punctuation Restoration for Turkish
Explainability of Speech Recognition Transformers via Gradient-based Attention Visualization
Building a Public Domain Voice Database for Odia
Automatic Speech Recognition in German: A Detailed Error Analysis
Which French speech recognition system for assistant robots?
토론
이 논문과 연결된 포럼 글이 아직 없습니다.