Structural Knowledge Distillation for Object Detection
원문 초록
Knowledge Distillation (KD) is a well-known training paradigm in deep neural networks where knowledge acquired by a large teacher model is transferred to a small student. KD has proven to be an effective technique to significantly improve the student's performance for various tasks including object detection. As such, KD techniques mostly rely on guidance at the intermediate feature level, which is typically implemented by minimizing an lp-norm distance between teacher and student activations during training. In this paper, we propose a replacement for the pixel-wise independent lp-norm based on the structural similarity (SSIM). By taking into account additional contrast and structural cues, feature importance, correlation and spatial dependence in the feature space are considered in the loss formulation. Extensive experiments on MSCOCO demonstrate the effectiveness of our method across different training schemes and architectures. Our method adds only little computational overhead, is straightforward to implement and at the same time it significantly outperforms the standard lp-norms. Moreover, more complex state-of-the-art KD methods using attention-based sampling mechanisms are outperformed, including a +3.5 AP gain using a Faster R-CNN R-50 compared to a vanilla model.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Towards Reliable Detection of Empty Space: Conditional Marked Point Processes for Object Detection
A Real-Time System for Egocentric Hand-Object Interaction Detection in Industrial Domains
ECORE: Energy-Conscious Optimized Routing for Deep Learning Models at the Edge
DuET: Dual Incremental Object Detection via Exemplar-Free Task Arithmetic
Class Agnostic Instance-level Descriptor for Visual Instance Search
Retrospective Memory for Camouflaged Object Detection
토론
이 논문과 연결된 포럼 글이 아직 없습니다.