A Comprehensive Solution to Connect Speech Encoder and Large Language Model for ASR
원문 초록
Recent works have shown promising results in connecting speech encoders to large language models (LLMs) for speech recognition. However, several limitations persist, including limited fine-tuning options, a lack of mechanisms to enforce speech-text alignment, and high insertion errors especially in domain mismatch conditions. This paper presents a comprehensive solution to address these issues. We begin by investigating more thoughtful fine-tuning schemes. Next, we propose a matching loss to enhance alignment between modalities. Finally, we explore training and inference methods to mitigate high insertion errors. Experimental results on the Librispeech corpus demonstrate that partially fine-tuning the encoder and LLM using parameter-efficient methods, such as LoRA, is the most cost-effective approach. Additionally, the matching loss improves modality alignment, enhancing performance. The proposed training and inference methods significantly reduce insertion errors.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition
Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR
LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization
Seewo's Submission to MLC-SLM: Lessons learned from Speech Reasoning Language Models
NTU Speechlab LLM-Based Multilingual ASR System for Interspeech MLC-SLM Challenge 2025
Qwen vs. Gemma Integration with Whisper: A Comparative Study in Multilingual SpeechLLM Systems
토론
이 논문과 연결된 포럼 글이 아직 없습니다.