Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
원문 초록
Self-supervised pre-training is an effective approach to leveraging a large amount of unlabelled data to reduce word error rates (WERs) of automatic speech recognition (ASR) systems. Since it is impractical to use large pre-trained models for many real-world ASR applications, it is desirable to have a much smaller model while retaining the performance of the pre-trained model. In this paper, we propose a simple knowledge distillation (KD) loss function for neural transducers that focuses on the one-best path in the output probability lattice under both streaming and non-streaming setups, which allows a small student model to approach the performance of the large pre-trained teacher model. Experiments on the LibriSpeech dataset show that despite being 10 times smaller than the teacher model, the proposed loss results in relative WER reductions (WERRs) of 11.5% and 6.8% on the test-other set for non-streaming and streaming student models compared to the baseline transducers trained without KD using the labelled 100-hour clean data. With an additional 860 hours of unlabelled data for KD, the WERRs increase to 48.2% and 38.5% for non-streaming and streaming students. If language model shallow fusion is used for producing distillation targets, a further improvement in the student model is observed.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Mlphon: A Multifunctional Grapheme-Phoneme Conversion Tool Using Finite State Transducers
Transformer Based Punctuation Restoration for Turkish
Explainability of Speech Recognition Transformers via Gradient-based Attention Visualization
Building a Public Domain Voice Database for Odia
Automatic Speech Recognition in German: A Detailed Error Analysis
Which French speech recognition system for assistant robots?
토론
이 논문과 연결된 포럼 글이 아직 없습니다.