Hierarchical Cross-Modality Knowledge Transfer with Sinkhorn Attention for CTC-based ASR
원문 초록
Due to the modality discrepancy between textual and acoustic modeling, efficiently transferring linguistic knowledge from a pretrained language model (PLM) to acoustic encoding for automatic speech recognition (ASR) still remains a challenging task. In this study, we propose a cross-modality knowledge transfer (CMKT) learning framework in a temporal connectionist temporal classification (CTC) based ASR system where hierarchical acoustic alignments with the linguistic representation are applied. Additionally, we propose the use of Sinkhorn attention in cross-modality alignment process, where the transformer attention is a special case of this Sinkhorn attention process. The CMKT learning is supposed to compel the acoustic encoder to encode rich linguistic knowledge for ASR. On the AISHELL-1 dataset, with CTC greedy decoding for inference (without using any language model), we achieved state-of-the-art performance with 3.64% and 3.94% character error rates (CERs) for the development and test sets, which corresponding to relative improvements of 34.18% and 34.88% compared to the baseline CTC-ASR system, respectively.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Mlphon: A Multifunctional Grapheme-Phoneme Conversion Tool Using Finite State Transducers
Transformer Based Punctuation Restoration for Turkish
Explainability of Speech Recognition Transformers via Gradient-based Attention Visualization
Building a Public Domain Voice Database for Odia
Automatic Speech Recognition in German: A Detailed Error Analysis
Which French speech recognition system for assistant robots?
토론
이 논문과 연결된 포럼 글이 아직 없습니다.