Integration of Large Language Models (LLMs) into visual domain tasks, resulting in visual-LLMs (V-LLMs), has enabled exceptional performance in vision-language tasks, particularly for visual question answering (VQA). However, existing V-LLMs (e.g. BLIP-2, LLaVA) demonstrate weak spatial reasoning and localization awareness. Despite generating highly descriptive and elaborate textual answers, these models fail at simple tasks like distinguishing a left vs right location. In this work, we explore how image-space coordinate based instruction fine-tuning objectives could inject spatial awareness into V-LLMs. We discover optimal coordinate representations, data-efficient instruction fine-tuning objectives, and pseudo-data generation strategies that lead to improved spatial awareness in V-LLMs. Additionally, our resulting model improves VQA across image and video domains, reduces undesired hallucination, and generates better contextual object descriptions. Experiments across 5 vision-language tasks involving 14 different datasets establish the clear performance improvements achieved by our proposed framework.
The development of 3D medical vision-language models holds significant potential for disease diagnosis and patient treatment. However, compared to 2D medical images, 3D medical images, such as CT scans, face challenges related to limited training data and high dimension, which severely restrict the progress of 3D medical vision-language models. To address these issues, we collect a large amount of unlabeled 3D CT data and utilize self-supervised learning to construct a 3D visual foundation model for extracting 3D visual features. Then, we apply 3D spatial convolutions to aggregate and project high-level image features, reducing computational complexity while preserving spatial information. We also construct two instruction-tuning datasets based on BIMCV-R and CT-RATE to fine-tune the 3D vision-language model. Our model demonstrates superior performance compared to existing methods in report generation, visual question answering, and disease diagnosis. Code and data will be made publicly available soon.
Multimodel Large Language Models(MLLMs) have achieved promising OCR-free Document Understanding performance by increasing the supported resolution of document images. However, this comes at the cost of generating thousands of visual tokens for a single document image, leading to excessive GPU memory and slower inference times, particularly in multi-page document comprehension. In this work, to address these challenges, we propose a High-resolution DocCompressor module to compress each high-resolution document image into 324 tokens, guided by low-resolution global visual features. With this compression module, to strengthen multi-page document comprehension ability and balance both token efficiency and question-answering performance, we develop the DocOwl2 under a three-stage training framework: Single-image Pretraining, Multi-image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new state-of-the-art across multi-page document understanding benchmarks and reduces first token latency by more than 50%, demonstrating advanced capabilities in multi-page questioning answering, explanation with evidence pages, and cross-page structure understanding. Additionally, compared to single-image MLLMs trained on similar data, our DocOwl2 achieves comparable single-page understanding performance with less than 20% of the visual tokens. Our codes, models, and data are publicly available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2.
종단적(longitudinal) 의료 시각 질의응답(VQA)은 현재 시점 이미지와 참조 시점 이미지 간의 해부학적 차이를 추론하는 과제입니다. 우리는 흉부 X선을 대상으로 이 과제를 위한 어텐션 기반 인코더-디코더를 제안합니다. 기존의 직접적인 대비 방식 대신, 소규모 등록 정규화기를 갖춘 경량 어파인(affine) 등록 모듈을 도입하여 현재 이미지를 참조 이미지에 공동 정합(co-registration)함으로써 불필요한 움직임을 줄입니다. 정합된 이미지 쌍은 이미지 인코더에 입력된 후, 고정된 DINO 기반 마스크 생성기와 학습 가능한 적응형 마스크 생성기를 거쳐 원본 이미지 쌍에 적용될 마스크를 생성합니다. 마스킹된 이미지 쌍은 다시 이미지 인코더에 입력되고, 텍스트 특징과 연결되어 멀티모달 트랜스포머 기반 디코더의 입력으로 활용되어 최종 답변을 생성합니다. DINO-v3에서 영감을 받아 학습 안정화 및 변화 신호 명확화를 위해 마스크 재구성 손실, 쌍별 Gram 스타일 일관성 손실, KoLeo 균등성 손실 등의 추가 보조 목적함수를 포함하여 표현의 기하학적 구조를 향상시킵니다. Medical-Diff-VQA 벤치마크에서 본 모델은 높은 BLEU, ROUGE-L, CIDEr, METEOR 점수를 달성하는 동시에 공유 현저성(saliency) 마스크를 통한 내재적 해석 가능성을 제공합니다. 이러한 결과는 경미한 사전 정합과 함께 현저성 조건부 생성이 의료 VQA의 종단적 추론을 위한 원칙적인 프레임워크임을 뒷받침합니다. 또한 우리의 학습 전략은 바이오메디컬 분야에서 이미지 기반 모델을 활용하는 패러다임의 가능성, 즉 지도 학습과 비지도 학습 목적함수를 동시에 최적화하는 방식의 잠재력을 보여줍니다.
Chart question answering (CQA) has become a critical multimodal task for evaluating the reasoning capabilities of vision-language models. While early approaches have shown promising performance by focusing on visual features or leveraging large-scale pre-training, most existing evaluations rely on rigid output formats and objective metrics, thus ignoring the complex, real-world demands of practical chart analysis. In this paper, we introduce ChartMind, a new benchmark designed for complex CQA tasks in real-world settings. ChartMind covers seven task categories, incorporates multilingual contexts, supports open-domain textual outputs, and accommodates diverse chart formats, bridging the gap between real-world applications and traditional academic benchmarks. Furthermore, we propose a context-aware yet model-agnostic framework, ChartLLM, that focuses on extracting key contextual elements, reducing noise, and enhancing the reasoning accuracy of multimodal large language models. Extensive evaluations on ChartMind and three representative public benchmarks with 14 mainstream multimodal models show our framework significantly outperforms the previous three common CQA paradigms: instruction-following, OCR-enhanced, and chain-of-thought, highlighting the importance of flexible chart understanding for real-world CQA. These findings suggest new directions for developing more robust chart reasoning in future research.
In contrast to conventional visual question answering, video-grounded dialog necessitates a profound understanding of both dialog history and video content for accurate response generation. Despite commendable progress made by existing approaches, they still face the challenges of incrementally understanding complex dialog history and assimilating video information. In response to these challenges, we present an iterative search and reasoning framework, which consists of a textual encoder, a visual encoder, and a generator. Specifically, we devise a path search and aggregation strategy in the textual encoder, mining core cues from dialog history that are pivotal to understanding the posed questions. Concurrently, our visual encoder harnesses an iterative reasoning network to extract and emphasize critical visual markers from videos, enhancing the depth of visual comprehension. Finally, we utilize the pre-trained GPT-2 model as our answer generator to decode the mined hidden clues into coherent and contextualized answers. Extensive experiments on three public datasets demonstrate the effectiveness and generalizability of our proposed framework.
Adversarial attacks are known to succeed on classifiers, but it has been an open question whether more complex vision systems are vulnerable. In this paper, we study adversarial examples for vision and language models, which incorporate natural language understanding and complex structures such as attention, localization, and modular architectures. In particular, we investigate attacks on a dense captioning model and on two visual question answering (VQA) models. Our evaluation shows that we can generate adversarial examples with a high success rate (i.e., > 90%) for these models. Our work sheds new light on understanding adversarial attacks on vision systems which have a language component and shows that attention, bounding box localization, and compositional internal structures are vulnerable to adversarial attacks. These observations will inform future work towards building effective defenses.
범용 구현형 에이전트(embodied agent)는 객체 인식 그 이상이 필요합니다. 이들은 상황화된 시각적 관찰로부터 공간 관계, 행동, 절차, 인간 의도, 환경적 제약, 그리고 상식적 결과에 대해 추론해야 합니다. 그러나 기존의 시각 및 구현형 질의응답 벤치마크는 테스트되는 추론 의존성에 대한 제어가 제한적인 경우가 많아, 그라운딩된(grounded) 구현형 추론과 지름길(shortcut) 기반의 시각적 또는 언어적 패턴 매칭을 구별하기 어렵습니다. 본 논문에서는 구현형 AI(Embodied AI)에서의 추론을 위한 진단용 벤치마크인 ERQA-Plus를 소개합니다. ERQA-Plus는 711개의 로봇 중심 이미지에 기반한 1,766개의 질문-답변 인스턴스를 포함하며, 지각적, 행동 중심, 사회적 상호작용, 네비게이션-환경, 그리고 맥락적 상식 추론에 걸친 구조화된 분류 체계에 따라 구성되었습니다. 이 데이터셋은 시각적 그라운딩(grounding), 답변 타당성, 그리고 추론 품질을 향상시키기 위해 분류 체계 기반 질문 생성, 자동 품질 판단, 반복적 수정, 그리고 인간 평가를 결합한 다단계 생성 및 검증 파이프라인을 사용하여 구축되었습니다. 우리는 LLaVA-NeXT-8B, Prismatic-7B, MiniCPM-V-4.5-8B, Qwen3-VL, RoboRefer-8B, RoboBrain2.5-8B를 포함한 대표적인 범용 비전-언어 모델(vision-language model)과 구현형 모델을 벤치마킹했습니다. 가장 강력한 모델인 Qwen3-VL-32B가 83.4%의 전체 정확도와 61.4의 SBERT 점수를 달성했지만, 카테고리 수준의 결과는 공간 추론, 절차적 추론, 사건 예측, 그리고 의도 추론에서 지속적인 약점을 보여줍니다. 따라서 ERQA-Plus는 구현형 에이전트가 올바르게 답변하는지 여부뿐만 아니라, 어떤 형태의 구현형 추론을 신뢰할 수 있게 수행할 수 있는지와 없는지를 측정하기 위한 세분화된 평가 프레임워크를 제공합니다. 데이터셋은 https://huggingface.co/datasets/huggingdas/erqa-plus 에서, 프로젝트 페이지는 https://github.com/LUNAProject22/erqa-plus 에서 확인할 수 있습니다.
무인 항공기(UAV) 비디오 질의응답은 카메라의 움직임과 장면의 변화를 분리하는 작업이 필요하지만, RGB 전용 멀티모달 모델은 이러한 분리를 위한 명시적이고 안정적인 기준을 제공받지 못합니다. 본 논문에서는 포즈(pose) 정보가 필요 없는 어댑터인 EgoSIS를 제안합니다. 이 어댑터는 RGB 기반 양방향 광학 흐름(flow)을 3단계에 거쳐 움직임 표준(motion-canonical) 시각 증거로 변환합니다. 첫째, 인수분해된 시각적 자기 전이(FVET)는 견고한 이미지 평면 전이를 피팅하고 움직임, 잔여 지지(residual-support), 신뢰도 요소를 도출합니다. 둘째, 신뢰도 게이트 자기 전이 메모리(ReTEM)는 제한된 이력에 대해 신뢰도 가중 업데이트를 사용하며, 컷 전환이나 지속적인 불확실성이 발생할 때 메모리를 재고정(re-anchor)합니다. 셋째, 자기 정렬 공간 증거(EASE)는 지지되는 시각 특징들을 각 세그먼트의 로컬 앵커로 워핑(warp)하고, Qwen의 시각 토큰(token) 수를 변경하지 않으면서 0으로 초기화된 잔여 연결을 통해 시각 슬라이스당 4개의 공간 증거 토큰을 주입합니다. SIS-Bench에서 EgoSIS-8B는 89.9%의 인지, 82.5%의 인지-메모리 결합, 76.2%의 종합 정확도를 달성했으며, 가장 큰 성능 향상은 자기 인식 인지 및 메모리에 집중되었습니다. 결과적으로 이 어댑터는 광학 흐름과 공간 추론 간의 해석 가능한 인터페이스를 제공합니다.
Deep neural networks continue to advance the state-of-the-art of image recognition tasks with various methods. However, applications of these methods to multimodality remain limited. We present Multimodal Residual Networks (MRN) for the multimodal residual learning of visual question-answering, which extends the idea of the deep residual learning. Unlike the deep residual learning, MRN effectively learns the joint representation from vision and language information. The main idea is to use element-wise multiplication for the joint residual mappings exploiting the residual learning of the attentional models in recent studies. Various alternative models introduced by multimodality are explored based on our study. We achieve the state-of-the-art results on the Visual QA dataset for both Open-Ended and Multiple-Choice tasks. Moreover, we introduce a novel method to visualize the attention effect of the joint representations for each learning block using back-propagation algorithm, even though the visual features are collapsed without spatial information.
We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.
Conversation agents powered by large language models are revolutionizing the way we interact with visual data. Recently, large vision-language models (LVLMs) have been extensively studied for both images and videos. However, these studies typically focus on common scenarios. In this work, we introduce an LVLM specifically designed for surgical scenarios. We integrate visual representations of surgical images and videos into the language feature space. Consequently, we establish a LVLM model, Surgical-LLaVA, fine-tuned on instruction following data of surgical scenarios. Our experiments demonstrate that Surgical-LLaVA exhibits impressive multi-modal chat abilities in surgical contexts, occasionally displaying multi-modal behaviors on unseen instructions. We conduct a quantitative evaluation of visual question-answering datasets for surgical scenarios. The results show superior performance compared to previous works, indicating the potential of our model to tackle more complex surgery scenarios.
Large vision-language models (LVLMs) have recently dramatically pushed the state of the art in image captioning and many image understanding tasks (e.g., visual question answering). LVLMs, however, often \textit{hallucinate} and produce captions that mention concepts that cannot be found in the image. These hallucinations erode the trustworthiness of LVLMs and are arguably among the main obstacles to their ubiquitous adoption. Recent work suggests that addition of grounding objectives -- those that explicitly align image regions or objects to text spans -- reduces the amount of LVLM hallucination. Although intuitive, this claim is not empirically justified as the reduction effects have been established, we argue, with flawed evaluation protocols that (i) rely on data (i.e., MSCOCO) that has been extensively used in LVLM training and (ii) measure hallucination via question answering rather than open-ended caption generation. In this work, in contrast, we offer the first systematic analysis of the effect of fine-grained object grounding on LVLM hallucination under an evaluation protocol that more realistically captures LVLM hallucination in open generation. Our extensive experiments over three backbone LLMs reveal that grounding objectives have little to no effect on object hallucination in open caption generation.
How to select relevant key objects and reason about the complex relationships cross vision and linguistic domain are two key issues in many multi-modality applications such as visual question answering (VQA). In this work, we incorporate the visual commonsense information and propose a heterogeneous graph contrastive learning method to better finish the visual reasoning task. Our method is designed as a plug-and-play way, so that it can be quickly and easily combined with a wide range of representative methods. Specifically, our model contains two key components: the Commonsense-based Contrastive Learning and the Graph Relation Network. Using contrastive learning, we guide the model concentrate more on discriminative objects and relevant visual commonsense attributes. Besides, thanks to the introduction of the Graph Relation Network, the model reasons about the correlations between homogeneous edges and the similarities between heterogeneous edges, which makes information transmission more effective. Extensive experiments on four benchmarks show that our method greatly improves seven representative VQA models, demonstrating its effectiveness and generalizability.
Large Language Models (LLMs) have shown remarkable performances on a wide range of natural language understanding and generation tasks. We observe that the LLMs provide effective priors in exploiting $\textit{linguistic shortcuts}$ for temporal and causal reasoning in Video Question Answering (VideoQA). However, such priors often cause suboptimal results on VideoQA by leading the model to over-rely on questions, $\textit{i.e.}$, $\textit{linguistic bias}$, while ignoring visual content. This is also known as `ungrounded guesses' or `hallucinations'. To address this problem while leveraging LLMs' prior on VideoQA, we propose a novel framework, Flipped-VQA, encouraging the model to predict all the combinations of $\langle$V, Q, A$\rangle$ triplet by flipping the source pair and the target label to understand their complex relationships, $\textit{i.e.}$, predict A, Q, and V given a VQ, VA, and QA pairs, respectively. In this paper, we develop LLaMA-VQA by applying Flipped-VQA to LLaMA, and it outperforms both LLMs-based and non-LLMs-based models on five challenging VideoQA benchmarks. Furthermore, our Flipped-VQA is a general framework that is applicable to various LLMs (OPT and GPT-J) and consistently improves their performances. We empirically demonstrate that Flipped-VQA not only enhances the exploitation of linguistic shortcuts but also mitigates the linguistic bias, which causes incorrect answers over-relying on the question. Code is available at https://github.com/mlvlab/Flipped-VQA.
Text-based VQA is a challenging task that requires machines to use scene texts in given images to yield the most appropriate answer for the given question. The main challenge of text-based VQA is exploiting the meaning and information from scene texts. Recent studies tackled this challenge by considering the spatial information of scene texts in images via embedding 2D coordinates of their bounding boxes. In this study, we follow the definition of meaning from linguistics to introduce a novel method that effectively exploits the information from scene texts written in Vietnamese. Experimental results show that our proposed method obtains state-of-the-art results on two large-scale Vietnamese Text-based VQA datasets. The implementation can be found at this link.
Previous studies such as VizWiz find that Visual Question Answering (VQA) systems that can read and reason about text in images are useful in application areas such as assisting visually-impaired people. TextVQA is a VQA dataset geared towards this problem, where the questions require answering systems to read and reason about visual objects and text objects in images. One key challenge in TextVQA is the design of a system that effectively reasons not only about visual and text objects individually, but also about the spatial relationships between these objects. This motivates the use of 'edge features', that is, information about the relationship between each pair of objects. Some current TextVQA models address this problem but either only use categories of relations (rather than edge feature vectors) or do not use edge features within the Transformer architectures. In order to overcome these shortcomings, we propose a Graph Relation Transformer (GRT), which uses edge information in addition to node information for graph attention computation in the Transformer. We find that, without using any other optimizations, the proposed GRT method outperforms the accuracy of the M4C baseline model by 0.65% on the val set and 0.57% on the test set. Qualitatively, we observe that the GRT has superior spatial reasoning ability to M4C.
Despite the existing evolution of Multimodal Large Language Models (MLLMs), a non-neglectable limitation remains in their struggle with visual text grounding, especially in text-rich images of documents. Document images, such as scanned forms and infographics, highlight critical challenges due to their complex layouts and textual content. However, current benchmarks do not fully address these challenges, as they mostly focus on visual grounding on natural images, rather than text-rich document images. Thus, to bridge this gap, we introduce TRIG, a novel task with a newly designed instruction dataset for benchmarking and improving the Text-Rich Image Grounding capabilities of MLLMs in document question-answering. Specifically, we propose an OCR-LLM-human interaction pipeline to create 800 manually annotated question-answer pairs as a benchmark and a large-scale training set of 90$ synthetic data based on four diverse datasets. A comprehensive evaluation of various MLLMs on our proposed benchmark exposes substantial limitations in their grounding capability on text-rich images. In addition, we propose two simple and effective TRIG methods based on general instruction tuning and plug-and-play efficient embedding, respectively. By finetuning MLLMs on our synthetic dataset, they promisingly improve spatial reasoning and grounding capabilities.
복잡한 시각 질의응답은 종종 비전-언어 모델이 이미지로 사고하는 것, 즉 고해상도 이미지의 정보가 풍부한 영역을 확대하여 국소적 시각 증거를 능동적으로 획득하는 것을 요구한다. 우리의 오류 분석에 따르면 이러한 설정에서 많은 실패는 잘못된 관찰 행동에서 비롯되며, 모델이 핵심 영역을 찾지 못하거나 답변에 충분한 증거를 담은 크롭을 얻지 못한다. 또한 바운딩 박스 토큰이 일반 언어 토큰보다 생성 엔트로피가 현저히 높다는 점을 관찰했으며, 이는 모델이 어디를 볼지 결정할 때 불확실성이 높음을 나타낸다. 그러나 최종 답변 보상에 기반한 기존 강화학습 방법은 궤적 수준의 피드백만 제공하므로 관찰 행동에 정확하게 기여도를 할당하기 어렵다. 우리는 도구를 활용하는 이미지로 사고하기에서 관찰 행동을 최적화하는 강화학습 알고리즘인 관찰 정책 최적화(OPO)를 제안한다. OPO는 두 가지 핵심 모듈로 구성된다. 첫째, 불확실성 유도 관찰 분기(UOB)는 각 이미지 확대 도구 호출을 관찰 이벤트로 취급하고 롤아웃 중 불확실성이 높고 중복성이 낮은 이벤트에서 선택적으로 분기하여, 대안적 크롭 결정과 그에 따른 시각 증거를 효율적으로 탐색한다. 둘째, 증거 국소화 이점 귀속(ELAA)은 국소적 증거 충분성 보상을 사용해 형제 분기들을 비교하고 해당 관찰 행동 토큰에만 기여도를 할당한다. 관찰 행동 최적화를 궤적 수준의 결과 지도와 분리함으로써 OPO는 시각 증거 획득을 개선하고 시각 추론 성능을 더욱 향상시킨다. 실험 결과 OPO는 다양한 규모의 Qwen3-VL 모델을 일관되게 개선하며, 더 큰 모델 및 도구 증강 기준선과 견줄 만한 성능을 달성한다. 코드는 공개되어 있다.
Traditional biomedical artificial intelligence (AI) models, designed for specific tasks or modalities, often exhibit limited flexibility in real-world deployment and struggle to utilize holistic information. Generalist AI holds the potential to address these limitations due to its versatility in interpreting different data types and generating tailored outputs for diverse needs. However, existing biomedical generalist AI solutions are typically heavyweight and closed source to researchers, practitioners, and patients. Here, we propose BiomedGPT, the first open-source and lightweight vision-language foundation model, designed as a generalist capable of performing various biomedical tasks. BiomedGPT achieved state-of-the-art results in 16 out of 25 experiments while maintaining a computing-friendly model scale. We also conducted human evaluations to assess the capabilities of BiomedGPT in radiology visual question answering, report generation, and summarization. BiomedGPT exhibits robust prediction ability with a low error rate of 3.8% in question answering, satisfactory performance with an error rate of 8.3% in writing complex radiology reports, and competitive summarization ability with a nearly equivalent preference score to human experts. Our method demonstrates that effective training with diverse data can lead to more practical biomedical AI for improving diagnosis and workflow efficiency.