MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation
원문 초록
Knowledge Distillation (KD) has been one of the most popu-lar methods to learn a compact model. However, it still suffers from highdemand in time and computational resources caused by sequential train-ing pipeline. Furthermore, the soft targets from deeper models do notoften serve as good cues for the shallower models due to the gap of com-patibility. In this work, we consider these two problems at the same time.Specifically, we propose that better soft targets with higher compatibil-ity can be generated by using a label generator to fuse the feature mapsfrom deeper stages in a top-down manner, and we can employ the meta-learning technique to optimize this label generator. Utilizing the softtargets learned from the intermediate feature maps of the model, we canachieve better self-boosting of the network in comparison with the state-of-the-art. The experiments are conducted on two standard classificationbenchmarks, namely CIFAR-100 and ILSVRC2012. We test various net-work architectures to show the generalizability of our MetaDistiller. Theexperiments results on two datasets strongly demonstrate the effective-ness of our method.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
A Unified Framework against Topology and Class Imbalance
Boosting Lightweight Single Image Super-resolution via Joint-distillation
DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition
Geo-ORBIT: A Federated Digital Twin Framework for Scene-Adaptive Lane Geometry Detection
The Trilemma of Truth in Large Language Models
G$^{2}$D: Boosting Multimodal Learning with Gradient-Guided Distillation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.