Vision-Language Models Do Not Understand Negation
원문 초록
Many practical vision-language applications require models that understand negation, e.g., when using natural language to retrieve images which contain certain objects but not others. Despite advancements in vision-language models (VLMs) through large-scale training, their ability to comprehend negation remains underexplored. This study addresses the question: how well do current VLMs understand negation? We introduce NegBench, a new benchmark designed to evaluate negation understanding across 18 task variations and $79$k examples spanning image, video, and medical datasets. The benchmark consists of two core tasks designed to evaluate negation understanding in diverse multimodal settings: Retrieval with Negation and Multiple Choice Questions with Negated Captions. Our evaluation reveals that modern VLMs struggle significantly with negation, often performing at chance level. To address these shortcomings, we explore a data-centric approach wherein we finetune CLIP models on large-scale synthetic datasets containing millions of negated captions. We show that this approach can result in a 10% increase in recall on negated queries and a 28% boost in accuracy on multiple-choice questions with negated captions.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding
Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic
Training-free LLM Merging for Multi-task Learning
STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean
토론
이 논문과 연결된 포럼 글이 아직 없습니다.