VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition
원문 초록
For visual recognition, knowledge distillation typically involves transferring knowledge from a large, well-trained teacher model to a smaller student model. In this paper, we introduce an effective method to distill knowledge from an off-the-shelf vision-language model (VLM), demonstrating that it provides novel supervision in addition to those from a conventional vision-only teacher model. Our key technical contribution is the development of a framework that generates novel text supervision and distills free-form text into a vision encoder. We showcase the effectiveness of our approach, termed VLM-KD, across various benchmark datasets, showing that it surpasses several state-of-the-art long-tail visual classifiers. To our knowledge, this work is the first to utilize knowledge distillation with text supervision generated by an off-the-shelf VLM and apply it to vanilla randomly initialized vision encoders.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
SeqPE: Transformer with Sequential Position Encoding
Visual-Language Model Knowledge Distillation Method for Image Quality Assessment
Label-Context-Dependent Internal Language Model Estimation for CTC
Induced Model Matching: Restricted Models Help Train Full-Featured Models
Disentangled Self-Supervision in Sequential Recommenders
Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks
토론
이 논문과 연결된 포럼 글이 아직 없습니다.