Assessing Open-world Forgetting in Generative Image Model Customization
원문 초록
Recent advances in diffusion models have significantly enhanced image generation capabilities. However, customizing these models with new classes often leads to unintended consequences that compromise their reliability. We introduce the concept of open-world forgetting to characterize the vast scope of these unintended alterations. Our work presents the first systematic investigation into open-world forgetting in diffusion models, focusing on semantic and appearance drift of representations. Using zero-shot classification, we demonstrate that even minor model adaptations can lead to significant semantic drift affecting areas far beyond newly introduced concepts, with accuracy drops of up to 60% on previously learned concepts. Our analysis of appearance drift reveals substantial changes in texture and color distributions of generated content. To address these issues, we propose a functional regularization strategy that effectively preserves original capabilities while accommodating new concepts. Through extensive experiments across multiple datasets and evaluation metrics, we demonstrate that our approach significantly reduces both semantic and appearance drift. Our study highlights the importance of considering open-world forgetting in future research on model customization and finetuning methods.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
DEARLi: Decoupled Enhancement of Recognition and Localization for Semi-supervised Panoptic Segmentation
Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models
GeoVision Labeler: Zero-Shot Geospatial Classification with Vision and Language Models
AmorLIP: Efficient Language-Image Pretraining via Amortization
Comparison of ConvNeXt and Vision-Language Models for Breast Density Assessment in Screening Mammography
Harmonizing and Merging Source Models for CLIP-based Domain Generalization
토론
이 논문과 연결된 포럼 글이 아직 없습니다.