In this paper, we introduce LLaVA-Octopus, a novel video multimodal large language model. LLaVA-Octopus adaptively weights features from different visual projectors based on user instructions, enabling us to leverage the complementary strengths of each projector. We observe that different visual projectors exhibit distinct characteristics when handling specific tasks. For instance, some projectors excel at capturing static details, while others are more effective at processing temporal information, and some are better suited for tasks requiring temporal coherence. By dynamically adjusting feature weights according to user instructions, LLaVA-Octopus dynamically selects and combines the most suitable features, significantly enhancing the model's performance in multimodal tasks. Experimental results demonstrate that LLaVA-Octopus achieves excellent performance across multiple benchmarks, especially in tasks such as multimodal understanding, visual question answering, and video understanding, highlighting its broad application potential.
Existing Large Vision-Language Models (LVLMs) excel at matching concepts across multi-modal inputs but struggle with compositional concepts and high-level relationships between entities. This paper introduces Progressive multi-granular Vision-Language alignments (PromViL), a novel framework to enhance LVLMs' ability in performing grounded compositional visual reasoning tasks. Our approach constructs a hierarchical structure of multi-modal alignments, ranging from simple to complex concepts. By progressively aligning textual descriptions with corresponding visual regions, our model learns to leverage contextual information from lower levels to inform higher-level reasoning. To facilitate this learning process, we introduce a data generation process that creates a novel dataset derived from Visual Genome, providing a wide range of nested compositional vision-language pairs. Experimental results demonstrate that our PromViL framework significantly outperforms baselines on various visual grounding and compositional question answering tasks. The code is available at: https://github.com/lqh52/PromViL.
To bridge the gap between vision and language modalities, Multimodal Large Language Models (MLLMs) usually learn an adapter that converts visual inputs to understandable tokens for Large Language Models (LLMs). However, most adapters generate consistent visual tokens, regardless of the specific objects of interest mentioned in the prompt. Since these adapters distribute equal attention to every detail in the image and focus on the entire scene, they may increase the cognitive load for LLMs, particularly when processing complex scenes. To alleviate this problem, we propose prompt-aware adapters. These adapters are designed with the capability to dynamically embed visual inputs based on the specific focus of the prompt. Specifically, prompt-aware adapters utilize both global and local textual features to capture the most relevant visual clues from the prompt at both coarse and fine granularity levels. This approach significantly enhances the ability of LLMs to understand and interpret visual content. Experiments on various visual question answering tasks, such as counting and position reasoning, demonstrate the effectiveness of prompt-aware adapters.
다중 페이지 문서 시각적 질의응답은 페이지 및 영역 수준 모두에서 희소한 증거를 찾아내는 것을 요구합니다. 기존 접근 방식들은 일반적으로 한 수준을 다른 수준보다 우선시합니다. 페이지 중심 방법은 페이지 획득에 초점을 맞추고 영역 작업은 주로 내비게이션 보조 수단으로 활용하는 반면, 영역 중심 방법은 관련 페이지가 이미 제공되었다고 가정합니다. 그 결과, 페이지 및 영역 선택은 연속적인 증거 결정 과정을 형성하지 못하고 단절된 상태로 남아 있습니다. 본 논문에서는 긴 문서의 증거 획득을 페이지에서 영역으로 이어지는 2단계 집합 예측으로 정형화하는 계층적 증거 라우팅 프레임워크인 HierDoc을 제안합니다. 페이지 정책은 전체 문서에서 증거 페이지를 선택하고, 이 페이지들은 의미적 요소 단위로 파싱된 후 영역 정책이 다운스트림 답변 모델로 전달할 요소를 선택합니다. 정답에 구애받지 않는 두 정책은 세분성 특화 구조화 집합 보상을 사용하는 단계별 GRPO로 최적화됩니다. 답변 모델은 선택된 전체 페이지와 함께 선택된 영역 크롭, 그리고 OCR 또는 표 텍스트를 전달받아 전역 문맥을 보존하면서 세밀한 증거를 강조합니다. 평가된 벤치마크 전반에서 HierDoc은 오픈 가중치 시스템 중 최고 수준(SOTA) 또는 경쟁력 있는 성능을 달성하여, 보고된 가장 강력한 오픈 가중치 기준선 대비 LongDocURL을 상대적으로 16.87% 향상시켰습니다. 통제된 소거 연구는 선택된 영역 증거가 페이지 전용 시스템의 정확도와 F1을 각각 5.51% 및 4.82% 향상시킴을 추가로 보여줍니다. 이러한 결과는 개략적인 페이지 라우팅과 세밀한 영역 라우팅을 통합 증거 획득 과정의 연속적이고 개별적으로 최적화된 단계로 구성하는 것의 이점을 입증합니다.
The transferability of adversarial examples is a crucial aspect of evaluating the robustness of deep learning systems, particularly in black-box scenarios. Although several methods have been proposed to enhance cross-model transferability, little attention has been paid to the transferability of adversarial examples across different tasks. This issue has become increasingly relevant with the emergence of foundational multi-task AI systems such as Visual ChatGPT, rendering the utility of adversarial samples generated by a single task relatively limited. Furthermore, these systems often entail inferential functions beyond mere recognition-like tasks. To address this gap, we propose a novel Visual Relation-based cross-task Adversarial Patch generation method called VRAP, which aims to evaluate the robustness of various visual tasks, especially those involving visual reasoning, such as Visual Question Answering and Image Captioning. VRAP employs scene graphs to combine object recognition-based deception with predicate-based relations elimination, thereby disrupting the visual reasoning information shared among inferential tasks. Our extensive experiments demonstrate that VRAP significantly surpasses previous methods in terms of black-box transferability across diverse visual reasoning tasks.
비전-언어 모델(VLM)은 이미지 캡셔닝(image captioning), 시각적 질의응답(visual question answering), 이미지-텍스트 생성(image-to-text generation)과 같은 과제에서 강한 성능을 달성했다. 그러나 이들은 주로 영어 중심 데이터로 학습되어, 문화적으로 그라운딩(grounding)된 시각 이해를 다루는 능력이 제한되며, 지역 특유의 의미, 상징적 내용, 맥락 의존적 시각 단서 해석에서 실패로 이어진다. 기존 문화적 역량 벤치마크는 종종 템플릿 기반이며 표면 수준 인식에 초점을 맞춰, 문화적 맥락이 주어진 환경에서 더 깊은 언어적·화용론적(pragmatic) 이해를 평가하기에 불충분하다. 우리는 PoVisLE를 제안한다. 이는 폴란드어를 위한 단일 문화(monocultural) 비전-언어 벤치마크로, 언어가 시각 맥락과의 상호작용 속에서 해석되는 그라운딩된 평가 패러다임(grounded evaluation paradigm) 하에서 문화적으로 그라운딩된 멀티모달 이해를 평가하도록 설계되었다. 데이터셋은 1,117개의 이미지와 2,366개의 수동 주석 VQA 쌍을 포함한다. 전반적으로, 본 데이터셋은 표면 수준 인식을 넘어선 문화적으로 그라운딩된 비전-언어 이해를 평가하기 위한 통제되고 도전적인 자원을 제공한다.
We perform a comprehensive benchmarking of contrastive frameworks for learning multimodal representations in the medical domain. Through this study, we aim to answer the following research questions: (i) How transferable are general-domain representations to the medical domain? (ii) Is multimodal contrastive training sufficient, or does it benefit from unimodal training as well? (iii) What is the impact of feature granularity on the effectiveness of multimodal medical representation learning? To answer these questions, we investigate eight contrastive learning approaches under identical training setups, and train them on 2.8 million image-text pairs from four datasets, and evaluate them on 25 downstream tasks, including classification (zero-shot and linear probing), image-to-text and text-to-image retrieval, and visual question-answering. Our findings suggest a positive answer to the first question, a negative answer to the second question, and the benefit of learning fine-grained features. Finally, we make our code publicly available.
Automatic math problem solving has recently attracted increasing attention as a long-standing AI benchmark. In this paper, we focus on solving geometric problems, which requires a comprehensive understanding of textual descriptions, visual diagrams, and theorem knowledge. However, the existing methods were highly dependent on handcraft rules and were merely evaluated on small-scale datasets. Therefore, we propose a Geometric Question Answering dataset GeoQA, containing 4,998 geometric problems with corresponding annotated programs, which illustrate the solving process of the given problems. Compared with another publicly available dataset GeoS, GeoQA is 25 times larger, in which the program annotations can provide a practical testbed for future research on explicit and explainable numerical reasoning. Moreover, we introduce a Neural Geometric Solver (NGS) to address geometric problems by comprehensively parsing multimodal information and generating interpretable programs. We further add multiple self-supervised auxiliary tasks on NGS to enhance cross-modal semantic representation. Extensive experiments on GeoQA validate the effectiveness of our proposed NGS and auxiliary tasks. However, the results are still significantly lower than human performance, which leaves large room for future research. Our benchmark and code are released at https://github.com/chen-judge/GeoQA .
Charts are ubiquitous, as people often use them to analyze data, answer questions, and discover critical insights. However, performing complex analytical tasks with charts requires significant perceptual and cognitive effort. Chart Question Answering (CQA) systems automate this process by enabling models to interpret and reason with visual representations of data. However, existing benchmarks like ChartQA lack real-world diversity and have recently shown performance saturation with modern large vision-language models (LVLMs). To address these limitations, we introduce ChartQAPro, a new benchmark that includes 1,341 charts from 157 diverse sources, spanning various chart types, including infographics and dashboards, and featuring 1,948 questions in various types, such as multiple-choice, conversational, hypothetical, and unanswerable questions, to better reflect real-world challenges. Our evaluations with 21 models show a substantial performance drop for LVLMs on ChartQAPro; e.g., Claude Sonnet 3.5 scores 90.5% on ChartQA but only 55.81% on ChartQAPro, underscoring the complexity of chart reasoning. We complement our findings with detailed error analyses and ablation studies, identifying key challenges and opportunities for advancing LVLMs in chart understanding and reasoning. We release ChartQAPro at https://github.com/vis-nlp/ChartQAPro.
비전-언어 모델(VLM)은 시각적 질의응답 벤치마크에서 강한 성능을 보이지만, 이미 올바르게 식별한 시각적 증거와 모순되는 결정을 내리는 경우가 종종 있다. 우리는 관련 증거가 인식되지 않는 지각 실패(perceptual failure)와, 인식된 증거가 최종 결정을 제약하지 못하는 과정 실패(process failure)를 구분한다. 우리는 VPAC-Bench를 제안한다. 이는 아홉 가지 실제 이미지 기반 과정(process) 계열을 아우르는 벤치마크로, 각 이미지에 현재 활동 단계(activity stage)와 인접한 단계 전환(stage transition)이 주석으로 달려 있다. 또한 State-Relevance-Target(SRT)를 제안한다. 이는 답변 전에 모델이 가시적 증거를 관련 과정 상태(process state)에 연결하도록 요구하는 구조화된 과정 사전(process-prior) 개입(intervention) 방법군이다. 여러 VLM에 걸쳐 과정 실패는 광범위하다. 시각적 후보를 올바르게 열거하는 모델조차 모호한 경우의 95% 이상에서 단일 답에 과도하게 고착(over-commit)한다. 명시적인 과정 구조화 개입은 모호하지 않은 경우의 성능을 저하시키지 않으면서 이 비율을 13% 미만으로 낮춘다. 그러나 과정 사전의 전이(transfer)는 모델에 따라 다르며, 일반적인 SRT는 강력한 사고의 연쇄(Chain-of-Thought) 베이스라인을 일관되게 능가하지 못한다. 관련 단계 전환이 알려져 있을 때, 경계 정렬(boundary-aligned) SRT는 조립(assembly), 물리적 상태 전환, 내비게이션 및 교통, 객체 사용 affordance 과제 전반에서 일반적인 과정 프롬프팅 및 검증된 모든 사고의 연쇄 베이스라인을 크게 능가한다. 이 결과는 과정 사전이 장면의 특정 의사결정 경계(decision boundary)와 정렬될 때 가장 유용함을 보여주며, 과정 기반(process-grounded) 시각 추론을 위한 경계 인식(boundary-aware) 사전 선택의 필요성을 뒷받침한다.
Vision Language Models (VLMs) extend remarkable capabilities of text-only large language models and vision-only models, and are able to learn from and process multi-modal vision-text input. While modern VLMs perform well on a number of standard image classification and image-text matching tasks, they still struggle with a number of crucial vision-language (VL) reasoning abilities such as counting and spatial reasoning. Moreover, while they might be very brittle to small variations in instructions and/or evaluation protocols, existing benchmarks fail to evaluate their robustness (or rather the lack of it). In order to couple challenging VL scenarios with comprehensive robustness evaluation, we introduce DARE, Diverse Visual Question Answering with Robustness Evaluation, a carefully created and curated multiple-choice VQA benchmark. DARE evaluates VLM performance on five diverse categories and includes four robustness-oriented evaluations based on the variations of: prompts, the subsets of answer options, the output format and the number of correct answers. Among a spectrum of other findings, we report that state-of-the-art VLMs still struggle with questions in most categories and are unable to consistently deliver their peak performance across the tested robustness evaluations. The worst case performance across the subsets of options is up to 34% below the performance in the standard case. The robustness of the open-source VLMs such as LLaVA 1.6 and Idefics2 cannot match the closed-source models such as GPT-4 and Gemini, but even the latter remain very brittle to different variations.
In this paper, we consider the problem of solving semantic tasks such as `Visual Question Answering' (VQA), where one aims to answers related to an image and `Visual Question Generation' (VQG), where one aims to generate a natural question pertaining to an image. Solutions for VQA and VQG tasks have been proposed using variants of encoder-decoder deep learning based frameworks that have shown impressive performance. Humans however often show generalization by relying on exemplar based approaches. For instance, the work by Tversky and Kahneman suggests that humans use exemplars when making categorizations and decisions. In this work, we propose the incorporation of exemplar based approaches towards solving these problems. Specifically, we incorporate exemplar based approaches and show that an exemplar based module can be incorporated in almost any of the deep learning architectures proposed in the literature and the addition of such a block results in improved performance for solving these tasks. Thus, just as the incorporation of attention is now considered de facto useful for solving these tasks, similarly, incorporating exemplars also can be considered to improve any proposed architecture for solving this task. We provide extensive empirical analysis for the same through various architectures, ablations, and state of the art comparisons.
Dialog systems need to understand dynamic visual scenes in order to have conversations with users about the objects and events around them. Scene-aware dialog systems for real-world applications could be developed by integrating state-of-the-art technologies from multiple research areas, including: end-to-end dialog technologies, which generate system responses using models trained from dialog data; visual question answering (VQA) technologies, which answer questions about images using learned image features; and video description technologies, in which descriptions/captions are generated from videos using multimodal information. We introduce a new dataset of dialogs about videos of human behaviors. Each dialog is a typed conversation that consists of a sequence of 10 question-and-answer(QA) pairs between two Amazon Mechanical Turk (AMT) workers. In total, we collected dialogs on roughly 9,000 videos. Using this new dataset for Audio Visual Scene-aware dialog (AVSD), we trained an end-to-end conversation model that generates responses in a dialog about a video. Our experiments demonstrate that using multimodal features that were developed for multimodal attention-based video description enhances the quality of generated dialog about dynamic scenes (videos). Our dataset, model code and pretrained models will be publicly available for a new Video Scene-Aware Dialog challenge.
We present UniFluid, a unified autoregressive framework for joint visual generation and understanding leveraging continuous visual tokens. Our unified autoregressive architecture processes multimodal image and text inputs, generating discrete tokens for text and continuous tokens for image. We find though there is an inherent trade-off between the image generation and understanding task, a carefully tuned training recipe enables them to improve each other. By selecting an appropriate loss balance weight, the unified model achieves results comparable to or exceeding those of single-task baselines on both tasks. Furthermore, we demonstrate that employing stronger pre-trained LLMs and random-order generation during training is important to achieve high-fidelity image generation within this unified framework. Built upon the Gemma model series, UniFluid exhibits competitive performance across both image generation and understanding, demonstrating strong transferability to various downstream tasks, including image editing for generation, as well as visual captioning and question answering for understanding.
Visual Question Answering (VQA) models often perform poorly on out-of-distribution data and struggle on domain generalization. Due to the multi-modal nature of this task, multiple factors of variation are intertwined, making generalization difficult to analyze. This motivates us to introduce a virtual benchmark, Super-CLEVR, where different factors in VQA domain shifts can be isolated in order that their effects can be studied independently. Four factors are considered: visual complexity, question redundancy, concept distribution and concept compositionality. With controllably generated data, Super-CLEVR enables us to test VQA methods in situations where the test data differs from the training data along each of these axes. We study four existing methods, including two neural symbolic methods NSCL and NSVQA, and two non-symbolic methods FiLM and mDETR; and our proposed method, probabilistic NSVQA (P-NSVQA), which extends NSVQA with uncertainty reasoning. P-NSVQA outperforms other methods on three of the four domain shift factors. Our results suggest that disentangling reasoning and perception, combined with probabilistic uncertainty, form a strong VQA model that is more robust to domain shifts. The dataset and code are released at https://github.com/Lizw14/Super-CLEVR.
Learning an effective attention mechanism for multimodal data is important in many vision-and-language tasks that require a synergic understanding of both the visual and textual contents. Existing state-of-the-art approaches use co-attention models to associate each visual object (e.g., image region) with each textual object (e.g., query word). Despite the success of these co-attention models, they only model inter-modal interactions while neglecting intra-modal interactions. Here we propose a general `unified attention' model that simultaneously captures the intra- and inter-modal interactions of multimodal features and outputs their corresponding attended representations. By stacking such unified attention blocks in depth, we obtain the deep Multimodal Unified Attention Network (MUAN), which can seamlessly be applied to the visual question answering (VQA) and visual grounding tasks. We evaluate our MUAN models on two VQA datasets and three visual grounding datasets, and the results show that MUAN achieves top-level performance on both tasks without bells and whistles.
Infographics are documents designed to effectively communicate information using a combination of textual, graphical and visual elements. In this work, we explore the automatic understanding of infographic images by using Visual Question Answering technique.To this end, we present InfographicVQA, a new dataset that comprises a diverse collection of infographics along with natural language questions and answers annotations. The collected questions require methods to jointly reason over the document layout, textual content, graphical elements, and data visualizations. We curate the dataset with emphasis on questions that require elementary reasoning and basic arithmetic skills. Finally, we evaluate two strong baselines based on state of the art multi-modal VQA models, and establish baseline performance for the new task. The dataset, code and leaderboard will be made available at http://docvqa.org
최근 몇 년간 멀티모달 대규모 언어 모델(MLLM)은 시각 질의응답 과제에서 크게 발전해 왔다. 그러나 기존 미세조정 방법을 원격 탐사(RS) 이미지에 그대로 적용하면 배경 잡음에 과적합되거나 대상의 세부 정보를 간과하는 문제가 자주 발생한다. 이는 주로 RS 이미지에 내재된 큰 규모 변화, 희소한 대상 분포, 복잡한 영역별 의미 특징 때문이다. 이러한 문제는 RS 과제에서 MLLM의 효과를 제한한다. 이를 해결하기 위해 본 논문은 안내형 영역 인식 희소 프롬프팅(GRASP)이라는 매개변수 효율적 미세조정(PEFT) 전략을 제안한다. GRASP는 고정된 시각 토큰 격자에서 추출한 공간 블록과 연관된 공간 구조의 소프트 프롬프트를 도입한다. 질문 안내형 희소 융합 메커니즘을 통해 GRASP는 과제별 문맥을 간결한 전역 프롬프트로 동적으로 집약하여, 모델이 배경 잡음을 걸러내면서 관련 영역에 집중하도록 한다. 여러 RSVQA 벤치마크에 대한 폭넓은 실험에서 GRASP는 높은 매개변수 효율을 유지하면서 기존 미세조정 및 프롬프트 기반 방법과 견줄 만한 성능을 달성하였다.
Multimodal latent-space reasoning aims to replace explicit thinking with images by performing visual reasoning directly in a compact latent space. However, existing approaches largely rely on visual supervision and produce latent representations that lack sufficient semantic richness, limiting their ability to support diverse region-level reasoning tasks. In this work, we introduce Semantic-Enriched Latent Visual Reasoning (SLVR), a two-stage learning framework that enriches latent representations with attribute-level visual semantics and aligns them with diverse reasoning objectives. In the first stage, SLVR learns semantically enriched region-centric latents under fine-grained attribute supervision. In the second stage, we design Multi-query Group Relative Policy Optimization (M-GRPO) to align latent representations across multiple queries grounded in the same region. To support this framework, we construct SLV-Set, comprising approximately 400K region-level attribute annotations and 800K multi-query question answering samples, and introduce SV-QA, a benchmark that evaluates latent reasoning under semantic variation. Experiments demonstrate that SLVR improves the robustness and semantic consistency of latent visual reasoning compared to existing baselines.
For visual document understanding (VDU), self-supervised pretraining has been shown to successfully generate transferable representations, yet, effective adaptation of such representations to distribution shifts at test-time remains to be an unexplored area. We propose DocTTA, a novel test-time adaptation method for documents, that does source-free domain adaptation using unlabeled target document data. DocTTA leverages cross-modality self-supervised learning via masked visual language modeling, as well as pseudo labeling to adapt models learned on a \textit{source} domain to an unlabeled \textit{target} domain at test time. We introduce new benchmarks using existing public datasets for various VDU tasks, including entity recognition, key-value extraction, and document visual question answering. DocTTA shows significant improvements on these compared to the source model performance, up to 1.89\% in (F1 score), 3.43\% (F1 score), and 17.68\% (ANLS score), respectively. Our benchmark datasets are available at \url{https://saynaebrahimi.github.io/DocTTA.html}.