Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
원문 초록
Large vision-language models (LVLMs) have made substantial progress in integrating large language models (LLMs) with visual inputs, enabling advanced multimodal reasoning. Despite their success, a persistent challenge is hallucination-where generated text fails to accurately reflect visual content-undermining both accuracy and reliability. Existing methods focus on alignment training or decoding refinements but primarily address symptoms at the generation stage without probing the underlying causes. In this work, we investigate the internal mechanisms driving hallucination in LVLMs, with an emphasis on the multi-head attention module. Specifically, we introduce Vision-aware Head Divergence (VHD), a metric that quantifies the sensitivity of attention head outputs to visual context. Based on this, our findings reveal the presence of vision-aware attention heads that are more attuned to visual information; however, the model's overreliance on its prior language patterns is closely related to hallucinations. Building on these insights, we propose Vision-aware Head Reinforcement (VHR), a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads. Extensive experiments demonstrate that our method achieves superior performance compared to state-of-the-art approaches in mitigating hallucinations, while maintaining high efficiency with negligible additional time overhead.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibration
DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning
VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM
Preemptive Hallucination Reduction: An Input-Level Approach for Multimodal Language Model
More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
토론
이 논문과 연결된 포럼 글이 아직 없습니다.