Knowledge Distillation via Instance-level Sequence Learning
원문 초록
Recently, distillation approaches are suggested to extract general knowledge from a teacher network to guide a student network. Most of the existing methods transfer knowledge from the teacher network to the student via feeding the sequence of random mini-batches sampled uniformly from the data. Instead, we argue that the compact student network should be guided gradually using samples ordered in a meaningful sequence. Thus, it can bridge the gap of feature representation between the teacher and student network step by step. In this work, we provide a curriculum learning knowledge distillation framework via instance-level sequence learning. It employs the student network of the early epoch as a snapshot to create a curriculum for the student network's next training phase. We carry out extensive experiments on CIFAR-10, CIFAR-100, SVHN and CINIC-10 datasets. Compared with several state-of-the-art methods, our framework achieves the best performance with fewer iterations.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
A Dual-Space Framework for General Knowledge Distillation of Large Language Models
Distilling Stereo Networks for Performant and Efficient Leaner Networks
SAM-Guided Robust Representation Learning for One-Shot 3D Medical Image Segmentation
Knowledge Distillation for Underwater Feature Extraction and Matching via GAN-synthesized Images
DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition
The Trilemma of Truth in Large Language Models
토론
이 논문과 연결된 포럼 글이 아직 없습니다.