CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs
원문 초록
Despite recent successes, LVLMs or Large Vision Language Models are prone to hallucinating details like objects and their properties or relations, limiting their real-world deployment. To address this and improve their robustness, we present CLIP-DPO, a preference optimization method that leverages contrastively pre-trained Vision-Language (VL) embedding models, such as CLIP, for DPO-based optimization of LVLMs. Unlike prior works tackling LVLM hallucinations, our method does not rely on paid-for APIs, and does not require additional training data or the deployment of other external LVLMs. Instead, starting from the initial pool of supervised fine-tuning data, we generate a diverse set of predictions, which are ranked based on their CLIP image-text similarities, and then filtered using a robust rule-based approach to obtain a set of positive and negative pairs for DPO-based training. We applied CLIP-DPO fine-tuning to the MobileVLM-v2 family of models and to LlaVA-1.5, in all cases observing significant improvements in terms of hallucination reduction over baseline models. We also observe better performance for zero-shot classification, suggesting improved grounding capabilities, and verify that the original performance on standard LVLM benchmarks is overall preserved.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
DEARLi: Decoupled Enhancement of Recognition and Localization for Semi-supervised Panoptic Segmentation
Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models
GeoVision Labeler: Zero-Shot Geospatial Classification with Vision and Language Models
AmorLIP: Efficient Language-Image Pretraining via Amortization
Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruption
StarFT: Robust Fine-tuning of Zero-shot Models via Spuriosity Alignment
토론
이 논문과 연결된 포럼 글이 아직 없습니다.