Enhanced Continual Learning of Vision-Language Models with Model Fusion
원문 초록
Vision-Language Models (VLMs) represent a breakthrough in artificial intelligence by integrating visual and textual modalities to achieve impressive zero-shot capabilities. However, VLMs are susceptible to catastrophic forgetting when sequentially fine-tuned on multiple downstream tasks. Existing continual learning methods for VLMs often rely heavily on additional reference datasets, compromise zero-shot performance, or are limited to parameter-efficient fine-tuning scenarios. In this paper, we propose Continual Decoupling-Unifying (ConDU), a novel approach, by introducing model fusion into continual learning for VLMs. ConDU maintains a unified model along with task triggers and prototype sets, employing an iterative process of decoupling task-specific models for previous tasks and unifying them with the model for the newly learned task. Additionally, we introduce an inference strategy for zero-shot scenarios by aggregating predictions from multiple decoupled task-specific models. Extensive experiments across various settings show that ConDU achieves up to a 2\% improvement in average performance across all seen tasks compared to state-of-the-art baselines, while also enhancing zero-shot capabilities relative to the original VLM.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning
Gated Integration of Low-Rank Adaptation for Continual Learning of Language Models
LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation
Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning
Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptation
Parameter-Efficient Continual Fine-Tuning: A Survey
토론
이 논문과 연결된 포럼 글이 아직 없습니다.