This paper investigates the robustness of vision-language models against adversarial visual perturbations and introduces a novel ``double visual defense" to enhance this robustness. Unlike previous approaches that resort to lightweight adversarial fine-tuning of a pre-trained CLIP model, we perform large-scale adversarial vision-language pre-training from scratch using web-scale data. We then strengthen the defense by incorporating adversarial visual instruction tuning. The resulting models from each stage, $\Delta$CLIP and $\Delta^2$LLaVA, show substantially enhanced zero-shot robustness and set a new state-of-the-art in adversarial defense for vision-language models. For example, the adversarial robustness of $\Delta$CLIP surpasses that of the previous best models on ImageNet-1k by ~20%. %For example, $\Delta$CLIP surpasses the previous best models on ImageNet-1k by ~20% in terms of adversarial robustness. Similarly, compared to prior art, $\Delta^2$LLaVA brings a ~30% robustness improvement to image captioning task and a ~20% robustness improvement to visual question answering task. Furthermore, our models exhibit stronger zero-shot recognition capability, fewer hallucinations, and superior reasoning performance compared to baselines. Our project page is https://doublevisualdefense.github.io/.
Document understanding aims to perform question answering and information extraction over document images, where the visual content is highly information-dense and most queries rely on only a few relevant layout regions. However, existing methods either adopt a one-pass strategy that implicitly assumes all layouts are equally important, or focus excessively on small regions at the cost of losing critical layout information. To address these limitations, we introduce Doc-CoB (Chain-of-Boxes), a simple-yet-effective framework that integrates coarse-to-fine layout-aware visual reasoning into multimodal large language models. Instead of directly zooming into small regions, Doc-CoB progressively focuses on query-relevant layouts while preserving global document information. Specifically, it first selects key layout boxes and then focuses on them for further understanding with visual prompting. To support this paradigm, we introduce two reasoning tasks for box recognition and box reasoning, with an automatic pipeline that constructs 249k training samples with intermediate visual supervision. Experiments on seven benchmarks with four popular models show that Doc-CoB significantly improves performance, demonstrating its effectiveness and wide applicability. The code and the data are available at https://github.com/Doc-CoB/Doc-CoB.
지식 기반 시각 질의응답(KB-VQA)은 Wikipedia 규모의 지식 베이스에서 검색한 구절을 리더(reader)에 조건으로 제공함으로써, 비전-언어 시스템이 매개변수적 지식을 넘어서는 질문에 답할 수 있게 한다. 순수 텍스트의 장문맥 LLM에서는 검색된 컨텍스트 활용이 Liu et al. (2024)의 U자형 "lost-in-the-middle" 효과를 따른다: 컨텍스트의 시작과 끝에 있는 정보는 사용되지만, 중간은 소실된다. 이 현상이 배포된 다중모달 KB-VQA로 전이되는지는 아직 열려 있다. 이 공백을 메우기 위해, 우리는 다중모달 KB-VQA에서 리더 측 위치 의존성을 최초로 통제적으로 탐색하는 프로브를 설계한다. 이는 질문 내에서 오직 정답(gold) 구절의 프롬프트 슬롯만 변화시키는 gold-position 프로토콜이다. 이를 세 개의 오픈소스 7B/8B VLM 리더와 두 개의 KB-VQA 벤치마크에서 k=20까지 실행한다. 효과의 형태는 U자형에서 초두 효과(primacy)로 뒤집힌다: gold-at-first는 모든 리더×벤치마크 조합에서 gold-at-last보다 16~26점 높으며, 우리는 이 효과를 "Lost at the End"라고 부른다. 세 가지 표적 ablation이 원인을 좁힌다: 텍스트 전용 통제는 다중모달 설정이 이미 존재하는 텍스트 모드의 초두 효과를 2.2~4.5배 증폭시킴을 보여주고, 이미지 위치 및 distractor 셔플 ablation을 함께 보면 원인은 instruction-tuned 리더의 프롬프트 슬롯 0에 있다. 고정(frozen) 리더에서는 세 가지 검색 측 수정(MMR, oracle reranking, rank-based reordering) 모두 격차를 그대로 남겨(분리 가능한 개선 없음) 둔다. 우리의 발견은 배포된 KB-VQA에서 재현율@k가 잘못된 지표임을, 그리고 격차를 줄이려면 리더 측 개입이 필요함을 시사한다. 우리는 이러한 개입을 평가하기 위한 통제된 도구로 해당 프로토콜을 공개한다.
The zero-shot performance of visual question answering (VQA) models relies heavily on prompts. For example, a zero-shot VQA for disaster scenarios could leverage well-designed Chain of Thought (CoT) prompts to stimulate the model's potential. However, using CoT prompts has some problems, such as causing an incorrect answer in the end due to the hallucination in the thought process. In this paper, we propose a zero-shot VQA named Flood Disaster VQA with Two-Stage Prompt (VQA-TSP). The model generates the thought process in the first stage and then uses the thought process to generate the final answer in the second stage. In particular, visual context is added in the second stage to relieve the hallucination problem that exists in the thought process. Experimental results show that our method exceeds the performance of state-of-the-art zero-shot VQA models for flood disaster scenarios in total. Our study provides a research basis for improving the performance of CoT-based zero-shot VQA.
Large Multimodal Models (LMMs) excel in natural language and visual understanding but are challenged by exacting tasks such as Knowledge-based Visual Question Answering (KB-VQA) which involve the retrieval of relevant information from document collections to use in shaping answers to questions. We present an extensive training and evaluation framework, M2KR, for KB-VQA. M2KR contains a collection of vision and language tasks which we have incorporated into a single suite of benchmark tasks for training and evaluating general-purpose multi-modal retrievers. We use M2KR to develop PreFLMR, a pre-trained version of the recently developed Fine-grained Late-interaction Multi-modal Retriever (FLMR) approach to KB-VQA, and we report new state-of-the-art results across a range of tasks. We also present investigations into the scaling behaviors of PreFLMR intended to be useful in future developments in general-purpose multi-modal retrievers.
With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks. However, the requirement for expensive annotations including clean image captions and regional labels limits the scalability of existing approaches, and complicates the pretraining procedure with the introduction of multiple dataset-specific objectives. In this work, we relax these constraints and present a minimalist pretraining framework, named Simple Visual Language Model (SimVLM). Unlike prior work, SimVLM reduces the training complexity by exploiting large-scale weak supervision, and is trained end-to-end with a single prefix language modeling objective. Without utilizing extra data or task-specific customization, the resulting model significantly outperforms previous pretraining methods and achieves new state-of-the-art results on a wide range of discriminative and generative vision-language benchmarks, including VQA (+3.74% vqa-score), NLVR2 (+1.17% accuracy), SNLI-VE (+1.37% accuracy) and image captioning tasks (+10.1% average CIDEr score). Furthermore, we demonstrate that SimVLM acquires strong generalization and transfer ability, enabling zero-shot behavior including open-ended visual question answering and cross-modality transfer.
Visual Entity Linking (VEL) is a task to link regions of images with their corresponding entities in Knowledge Bases (KBs), which is beneficial for many computer vision tasks such as image retrieval, image caption, and visual question answering. While existing tasks in VEL either rely on textual data to complement a multi-modal linking or only link objects with general entities, which fails to perform named entity linking on large amounts of image data. In this paper, we consider a purely Visual-based Named Entity Linking (VNEL) task, where the input only consists of an image. The task is to identify objects of interest (i.e., visual entity mentions) in images and link them to corresponding named entities in KBs. Since each entity often contains rich visual and textual information in KBs, we thus propose three different sub-tasks, i.e., visual to visual entity linking (V2VEL), visual to textual entity linking (V2TEL), and visual to visual-textual entity linking (V2VTEL). In addition, we present a high-quality human-annotated visual person linking dataset, named WIKIPerson. Based on WIKIPerson, we establish a series of baseline algorithms for the solution of each sub-task, and conduct experiments to verify the quality of proposed datasets and the effectiveness of baseline methods. We envision this work to be helpful for soliciting more works regarding VNEL in the future. The codes and datasets are publicly available at https://github.com/ict-bigdatalab/VNEL.
비전 언어 모델(VLM)은 3D 컴퓨터 단층촬영(CT) 분류에서 강력한 성능을 달성했지만, 높은 정확도가 예측이 올바른 해부학적 증거에 의존하는지를 반영하지는 않는다. 우리는 통제된 해부학적 개입을 통해 장기 수준의 영상-텍스트 정렬을 평가하고 개선하기 위한 프레임워크인 ORGANIC3D를 소개한다. 우리의 벤치마크는 짝지어진 환자 비교, 장기 제거, 병변 대체, 좌우 방향 교란을 이용하여 12개의 비전 인코더 체크포인트와 7개의 생성형 VLM을 평가하며, 최상위 성능 모델에서도 상당한 정렬 실패가 있음을 드러낸다. 장기 수준의 영상-텍스트 대응을 학습하기 위해 ORGANIC3D는 각 소견 문장을 해부학 인식 영상 특징과 정렬하고, 편집된 스캔으로 학습하여 이 정렬을 해당 장기의 영상 증거에 고정한다. 우리는 이 프레임워크를 생성형 VLM으로 확장하여 증거에 의존하는 질의응답과 보고서 생성을 촉진한다. 공개 데이터만으로 학습한 ORGANIC3D는 CTRATE, Merlin, 그리고 보류된 RADChestCT 데이터셋에서 SOTA 분류 AUROC를 달성하며, 기존 모델의 최대 66%에 비해 77%의 좌우 방향 정확도에 도달한다. 이 생성형 VLM은 시각적 질의응답에서 평가된 모든 공개 VLM과 동등하거나 이를 능가하고, 가장 높은 보고서 정확도를 달성하며, 장기 제거 후 대상 장기 소견의 96~99%를 철회하는 반면 다른 모델은 최대 69%이다. 이러한 결과는 종합적으로 ORGANIC3D가 3D CT 비전 언어 모델이 예측을 해부학적 증거에 근거하는지를 시험하고 예측 정확도와 함께 장기 수준의 정렬을 개선하기 위한 프레임워크임을 입증한다.
Medical Visual Question Answering (Med-VQA) systems benefit the interpretation of medical images containing critical clinical information. However, the challenge of noisy labels and limited high-quality datasets remains underexplored. To address this, we establish the first benchmark for noisy labels in Med-VQA by simulating human mislabeling with semantically designed noise types. More importantly, we introduce the DiN framework, which leverages a diffusion model to handle noisy labels in Med-VQA. Unlike the dominant classification-based VQA approaches that directly predict answers, our Answer Diffuser (AD) module employs a coarse-to-fine process, refining answer candidates with a diffusion model for improved accuracy. The Answer Condition Generator (ACG) further enhances this process by generating task-specific conditional information via integrating answer embeddings with fused image-question features. To address label noise, our Noisy Label Refinement(NLR) module introduces a robust loss function and dynamic answer adjustment to further boost the performance of the AD module.
Vision Language Models (VLMs), which extend Large Language Models (LLM) by incorporating visual understanding capability, have demonstrated significant advancements in addressing open-ended visual question-answering (VQA) tasks. However, these models cannot accurately interpret images infused with text, a common occurrence in real-world scenarios. Standard procedures for extracting information from images often involve learning a fixed set of query embeddings. These embeddings are designed to encapsulate image contexts and are later used as soft prompt inputs in LLMs. Yet, this process is limited to the token count, potentially curtailing the recognition of scenes with text-rich context. To improve upon them, the present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. BLIVA incorporates the query embeddings from InstructBLIP and also directly projects encoded patch embeddings into the LLM, a technique inspired by LLaVA. This approach assists the model to capture intricate details potentially missed during the query decoding process. Empirical evidence demonstrates that our model, BLIVA, significantly enhances performance in processing text-rich VQA benchmarks (up to 17.76% in OCR-VQA benchmark) and in undertaking general (not particularly text-rich) VQA benchmarks (up to 7.9% in Visual Spatial Reasoning benchmark), and achieved 17.72% overall improvement in a comprehensive multimodal LLM benchmark (MME), comparing to our baseline InstructBLIP. BLIVA demonstrates significant capability in decoding real-world images, irrespective of text presence. To demonstrate the broad industry applications enabled by BLIVA, we evaluate the model using a new dataset comprising YouTube thumbnails paired with question-answer sets across 11 diverse categories. Our code and models are freely accessible at https://github.com/mlpc-ucsd/BLIVA.
Deep models are the defacto standard in visual decision models due to their impressive performance on a wide array of visual tasks. However, they are frequently seen as opaque and are unable to explain their decisions. In contrast, humans can justify their decisions with natural language and point to the evidence in the visual world which led to their decisions. We postulate that deep models can do this as well and propose our Pointing and Justification (PJ-X) model which can justify its decision with a sentence and point to the evidence by introspecting its decision and explanation process using an attention mechanism. Unfortunately there is no dataset available with reference explanations for visual decision making. We thus collect two datasets in two domains where it is interesting and challenging to explain decisions. First, we extend the visual question answering task to not only provide an answer but also a natural language explanation for the answer. Second, we focus on explaining human activities which is traditionally more challenging than object classification. We extensively evaluate our PJ-X model, both on the justification and pointing tasks, by comparing it to prior models and ablations using both automatic and human evaluations.
Guessing games are a prototypical instance of the "learning by interacting" paradigm. This work investigates how well an artificial agent can benefit from playing guessing games when later asked to perform on novel NLP downstream tasks such as Visual Question Answering (VQA). We propose two ways to exploit playing guessing games: 1) a supervised learning scenario in which the agent learns to mimic successful guessing games and 2) a novel way for an agent to play by itself, called Self-play via Iterated Experience Learning (SPIEL). We evaluate the ability of both procedures to generalize: an in-domain evaluation shows an increased accuracy (+7.79) compared with competitors on the evaluation suite CompGuessWhat?!; a transfer evaluation shows improved performance for VQA on the TDIUC dataset in terms of harmonic average accuracy (+5.31) thanks to more fine-grained object representations learned via SPIEL.
지시적 의도는 표면적 형태만으로는 추론할 수 없기 때문에, 간접 화행(ISA)은 문맥에 대한 화용론적 추론을 필요로 한다. 이전의 텍스트 기반 연구와 기존 멀티모달 벤치마크는 이러한 요구사항을 대체로 간과하고 명시적으로 인코딩된 문맥이나 지각적 인식에만 집중하여, 특히 한국어와 같은 고맥락 언어에서 문맥 의존적인 화용론적 이해를 충분히 탐구하지 못했다. 우리는 시각적 문맥과 대화에 대한 통합적 추론을 통해 간접 화행 이해 능력을 평가하기 위한 멀티모달 벤치마크인 READI를 소개한다. READI는 화용론 이론에 기반하여 단계별 간접성을 모델링하고, 이 과제를 시각 기반 화용론적 질의응답(V-PQA)으로 정형화하여 영어와 한국어에 걸친 교차 언어 평가를 지원한다. 실험 결과에 따르면 최첨단 멀티모달 모델조차 시각적으로 그라운딩된 간접 화행을 처리하는 데 어려움을 겪으며, 간접성이 증가할수록 성능이 저하되는 것으로 나타나 문맥적 화용론 추론을 명시적으로 겨냥한 벤치마크의 필요성을 강조한다.
Text-rich VQA, namely Visual Question Answering based on text recognition in the images, is a cross-modal task that requires both image comprehension and text recognition. In this work, we focus on investigating the advantages and bottlenecks of LLM-based approaches in addressing this problem. To address the above concern, we separate the vision and language modules, where we leverage external OCR models to recognize texts in the image and Large Language Models (LLMs) to answer the question given texts. The whole framework is training-free benefiting from the in-context ability of LLMs. This pipeline achieved superior performance compared to the majority of existing Multimodal Large Language Models (MLLM) on four text-rich VQA datasets. Besides, based on the ablation study, we find that LLM brings stronger comprehension ability and may introduce helpful knowledge for the VQA problem. The bottleneck for LLM to address text-rich VQA problems may primarily lie in visual part. We also combine the OCR module with MLLMs and pleasantly find that the combination of OCR module with MLLM also works. It's worth noting that not all MLLMs can comprehend the OCR information, which provides insights into how to train an MLLM that preserves the abilities of LLM.
In this work, we explore a genre of puzzles ("image riddles") which involves a set of images and a question. Answering these puzzles require both capabilities involving visual detection (including object, activity recognition) and, knowledge-based or commonsense reasoning. We compile a dataset of over 3k riddles where each riddle consists of 4 images and a groundtruth answer. The annotations are validated using crowd-sourced evaluation. We also define an automatic evaluation metric to track future progress. Our task bears similarity with the commonly known IQ tasks such as analogy solving, sequence filling that are often used to test intelligence. We develop a Probabilistic Reasoning-based approach that utilizes probabilistic commonsense knowledge to answer these riddles with a reasonable accuracy. We demonstrate the results of our approach using both automatic and human evaluations. Our approach achieves some promising results for these riddles and provides a strong baseline for future attempts. We make the entire dataset and related materials publicly available to the community in ImageRiddle Website (http://bit.ly/22f9Ala).
Images reveal appearances, but counting often asks about physical objects. A single entity may be visible directly, in several reflections, or through glass, so detecting every visible occurrence does not determine how many entities are present. Existing visual counting benchmarks rarely separate these two operations. We introduce PhysCount, a benchmark for evaluating whether multimodal large language models (MLLMs) can distinguish what appears in an image from what physically exists in the scene. PhysCount contains 546 real-world images and 4,595 human-written questions organized into 651 target-specific groups. For each group, annotators locate the relevant appearances, label how each is observed, and identify appearances of the same entity. Four complementary question families test the enumeration of appearances, the enumeration of physical entities, image-formation-specific counting, and appearance--entity correspondence. We measure both accuracy on each family and the proportion of groups answered entirely correctly. Across 40 MLLMs, counting visible appearances is substantially easier than recovering the identities that connect them. Even the strongest model answers every question correctly in only 13.36\% of groups. Performance declines when one entity produces many appearances or when its appearances cross observation modes. Higher image resolution improves access to visual evidence but brings limited gains in complete-group accuracy, whereas answering related questions jointly produces a marked improvement. These results expose a gap between recognizing what is visible and correctly recovering the physical entities behind those observations.
In certain types of cancerous tissue, mitotic count has been shown to be associated with tumor proliferation, poor prognosis, and therapeutic resistance. Due to the high inter-rater variability of mitotic counting by pathologists, convolutional neural networks (CNNs) have been employed to reduce the subjectivity of mitosis detection in hematoxylin and eosin (H&E)-stained whole slide images. However, most existing models have performance that lags behind expert panel review and only incorporate visual information. In this work, we demonstrate that pre-trained large-scale vision-language models that leverage both visual features and natural language improve mitosis detection accuracy. We formulate the mitosis detection task as an image captioning task and a visual question answering (VQA) task by including metadata such as tumor and scanner types as context. The effectiveness of our pipeline is demonstrated via comparison with various baseline models using 9,501 mitotic figures and 11,051 hard negatives (non-mitotic figures that are difficult to characterize) from the publicly available Mitosis Domain Generalization Challenge (MIDOG22) dataset.
Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects
벵골(Bangla) 문화는 지역, 방언, 역사, 음식, 정치, 미디어, 일상적 시각적 삶을 통해 풍부하게 드러나지만, 멀티모달 평가에서는 여전히 과소 대표되어 있다. 이 공백을 메우기 위해, 역사적으로 연관된 언어와 지역 방언에 걸쳐 벵골 문화에 대한 다국어 비전-언어 모델(VLM)을 평가하기 위한 문화적으로 그라운딩된 벤치마크인 BanglaVerse를 제안한다. 9개 도메인에 걸친 1,152장의 수동 큐레이션 이미지를 바탕으로 구축된 이 벤치마크는 시각적 질의응답(VQA)과 캡셔닝을 지원하며, 4개 언어와 5개 벵골 방언으로 확장되어 약 32.2K개의 아티팩트를 산출한다. 실험 결과, 표준 벵골어만으로 평가하면 실제 모델 능력을 과대평가하는 것으로 나타났다. 성능은 방언 변이 하에서, 특히 캡션 생성에서 하락하며, 힌디어·우르두어와 같이 역사적으로 연관된 언어는 일부 문화적 의미는 유지하지만 구조적 추론에서는 여전히 약하다. 도메인 전반에서 주요 병목은 시각적 그라운딩만이 아니라 부재한 문화적 지식이며, 지식 집약적 카테고리에서 두드러진다. 이러한 결과는 BanglaVerse가 언어적 변이 하에서 문화적으로 그라운딩된 멀티모달 이해를 측정하기 위한 보다 현실적인 시험대임을 보여준다.
Automatic Chart Question Answering (ChartQA) is challenging due to the complex distribution of chart elements with patterns of the underlying data not explicitly displayed in charts. To address this challenge, we design a joint multimodal scene graph for charts to explicitly represent the relationships between chart elements and their patterns. Our proposed multimodal scene graph includes a visual graph and a textual graph to jointly capture the structural and semantical knowledge from the chart. This graph module can be easily integrated with different vision transformers as inductive bias. Our experiments demonstrate that incorporating the proposed graph module enhances the understanding of charts' elements' structure and semantics, thereby improving performance on publicly available benchmarks, ChartQA and OpenCQA.
The combination of visual and textual representations has produced excellent results in tasks such as image captioning and visual question answering, but the inference capabilities of multimodal representations are largely untested. In the case of textual representations, inference tasks such as Textual Entailment and Semantic Textual Similarity have been often used to benchmark the quality of textual representations. The long term goal of our research is to devise multimodal representation techniques that improve current inference capabilities. We thus present a novel task, Visual Semantic Textual Similarity (vSTS), where such inference ability can be tested directly. Given two items comprised each by an image and its accompanying caption, vSTS systems need to assess the degree to which the captions in context are semantically equivalent to each other. Our experiments using simple multimodal representations show that the addition of image representations produces better inference, compared to text-only representations. The improvement is observed both when directly computing the similarity between the representations of the two items, and when learning a siamese network based on vSTS training data. Our work shows, for the first time, the successful contribution of visual information to textual inference, with ample room for benchmarking more complex multimodal representation options.