SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning
원문 초록
Self-supervised learning (SSL) has achieved remarkable success across various speech-processing tasks. To enhance its efficiency, previous works often leverage the use of compression techniques. A notable recent attempt is DPHuBERT, which applies joint knowledge distillation (KD) and structured pruning to learn a significantly smaller SSL model. In this paper, we contribute to this research domain by introducing SKILL, a novel method that conducts distillation across groups of layers instead of distilling individual arbitrarily selected layers within the teacher network. The identification of the layers to distill is achieved through a hierarchical clustering procedure applied to layer similarity measures. Extensive experiments demonstrate that our distilled version of WavLM Base+ not only outperforms DPHuBERT but also achieves state-of-the-art results in the 30M parameters model class across several SUPERB tasks.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Efficient and Generalizable Speaker Diarization via Structured Pruning of Self-Supervised Models
Continual Self-Supervised Learning with Masked Autoencoders in Remote Sensing
Enhancing Few-shot Keyword Spotting Performance through Pre-Trained Self-supervised Speech Models
Fine-tune Before Structured Pruning: Towards Compact and Accurate Self-Supervised Models for Speaker Diarization
DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition
The Trilemma of Truth in Large Language Models
토론
이 논문과 연결된 포럼 글이 아직 없습니다.