Heterogeneous Federated Learning Using Knowledge Codistillation
원문 초록
Federated Averaging, and many federated learning algorithm variants which build upon it, have a limitation: all clients must share the same model architecture. This results in unused modeling capacity on many clients, which limits model performance. To address this issue, we propose a method that involves training a small model on the entire pool and a larger model on a subset of clients with higher capacity. The models exchange information bidirectionally via knowledge distillation, utilizing an unlabeled dataset on a server without sharing parameters. We present two variants of our method, which improve upon federated averaging on image classification and language modeling tasks. We show this technique can be useful even if only out-of-domain or limited in-domain distillation data is available. Additionally, the bi-directional knowledge distillation allows for domain transfer between the models when different pool populations introduce domain shift.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Visual-Language Model Knowledge Distillation Method for Image Quality Assessment
Induced Model Matching: Restricted Models Help Train Full-Featured Models
Disentangled Self-Supervision in Sequential Recommenders
Recurrent Highway Networks with Grouped Auxiliary Memory
Making Language Model a Hierarchical Classifier and Generator
InstructFLIP: Exploring Unified Vision-Language Model for Face Anti-spoofing
토론
이 논문과 연결된 포럼 글이 아직 없습니다.