Masked Audio Text Encoders are Effective Multi-Modal Rescorers
원문 초록
Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition (ASR) systems. In this work, we propose Masked Audio Text Encoder (MATE), a multi-modal masked language model rescorer which incorporates acoustic representations into the input space of MLM. We adopt contrastive learning for effectively aligning the modalities by learning shared representations. We show that using a multi-modal rescorer is beneficial for domain generalization of the ASR system when target domain data is unavailable. MATE reduces word error rate (WER) by 4%-16% on in-domain, and 3%-7% on out-of-domain datasets, over the text-only baseline. Additionally, with very limited amount of training data (0.8 hours), MATE achieves a WER reduction of 8%-23% over the first-pass baseline.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization
Seewo's Submission to MLC-SLM: Lessons learned from Speech Reasoning Language Models
Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR
Improving Named Entity Transcription with Contextual LLM-based Revision
Customizing Speech Recognition Model with Large Language Model Feedback
Mlphon: A Multifunctional Grapheme-Phoneme Conversion Tool Using Finite State Transducers
토론
이 논문과 연결된 포럼 글이 아직 없습니다.