Understanding dynamics from visual observations is a challenging problem that requires disentangling individual objects from the scene and learning their interactions. While recent object-centric models can successfully decompose a scene into objects, modeling their dynamics effectively still remains a challenge. We address this problem by introducing SlotFormer -- a Transformer-based autoregressive model operating on learned object-centric representations. Given a video clip, our approach reasons over object features to model spatio-temporal relationships and predicts accurate future object states. In this paper, we successfully apply SlotFormer to perform video prediction on datasets with complex object interactions. Moreover, the unsupervised SlotFormer's dynamics model can be used to improve the performance on supervised downstream tasks, such as Visual Question Answering (VQA), and goal-conditioned planning. Compared to past works on dynamics modeling, our method achieves significantly better long-term synthesis of object dynamics, while retaining high quality visual generation. Besides, SlotFormer enables VQA models to reason about the future without object-level labels, even outperforming counterparts that use ground-truth annotations. Finally, we show its ability to serve as a world model for model-based planning, which is competitive with methods designed specifically for such tasks.
Language-Image Pre-training has demonstrated promising results on zero-shot and few-shot downstream tasks by prompting visual models with natural language prompts. However, most recent studies only use a single prompt for tuning, neglecting the inherent step-to-step cognitive reasoning process that humans conduct in complex task settings, for example, when processing images from unfamiliar domains. Chain of Thought is a simple and effective approximation to human reasoning process and has been proven useful for natural language processing (NLP) tasks. Based on this cognitive intuition, we believe that conducting effective reasoning is also an important problem in visual tasks, and a chain of thought could be a solution to this problem. In this work, we propose a novel chain of thought prompt tuning for vision-language modeling. Extensive experiments show that our method not only generalizes better in image classification tasks, has greater transferability beyond a single dataset, and has stronger domain generalization performance, but also performs much better in imagetext retrieval and visual question answering, which require more reasoning capabilities. We are the first to successfully adapt chain-of-thought prompting that combines visual and textual embeddings. We will release our codes
LLaVA와 GPT-4V로 대표되는 대규모 비전-언어 모델(LVLM)은 뛰어난 성능을 보여 왔지만, 시각 입력은 여전히 적대적 공격(adversarial attack)에 취약하여 심각한 보안 위험을 초래한다. 기존 방어 방법은 주로 단일 과제 시나리오(예: 제로샷(zero-shot) 분류)를 대상으로 하며, 그 결과 다양한 멀티모달 과제 전반에 걸친 일반화 능력이 부족하다. 이 한계를 해결하기 위해, 우리는 두 모달리티의 시각적·의미적 지도 신호를 공동으로 최적화하여 모델 견고성을 강화하면서 여러 하위 과제에 일반화하는 이중 적대적 미세조정(dual adversarial fine-tuning) 프레임워크를 제안한다. 제안 프레임워크는 $\textbf{Visual}$ 지도 분기와 $\textbf{Semantic}$ 지도 분기라는 두 핵심 구성 요소로 이루어진다. 전자는 동결된 원본 비전 인코더를 통해 추출한 깨끗한 이미지의 특징을 활용하여 적대적 견고성을 유도하고, 후자는 캡션-이미지 정렬(caption-image alignment)을 문맥적 신호로 통합하여 공격 상황에서도 의미적 일관성을 유지한다. 또한 우리 방법은 원본 모델의 CLIP 비전 인코더를 단순히 교체함으로써 과제별 재학습이나 아키텍처 수정 없이 교차 과제 견고성을 달성한다. 광범위한 실험 결과, 우리 접근법은 제로샷 분류, 이미지 캡셔닝(image captioning), 시각적 질의응답(VQA) 과제 전반의 적대적 견고성 평가에서 최신(state-of-the-art) 방법을 능가함을 보여준다.
Understanding the mechanisms of information storage and transfer in Transformer-based models is important for driving model understanding progress. Recent work has studied these mechanisms for Large Language Models (LLMs), revealing insights on how information is stored in a model's parameters and how information flows to and from these parameters in response to specific prompts. However, these studies have not yet been extended to Multi-modal Large Language Models (MLLMs). Given their expanding capabilities and real-world use, we start by studying one aspect of these models -- how MLLMs process information in a factual visual question answering task. We use a constraint-based formulation which views a visual question as having a set of visual or textual constraints that the model's generated answer must satisfy to be correct (e.g. What movie directed by the director in this photo has won a Golden Globe?). Under this setting, we contribute i) a method that extends causal information tracing from pure language to the multi-modal setting, and ii) VQA-Constraints, a test-bed of 9.7K visual questions annotated with constraints. We use these tools to study two open-source MLLMs, LLaVa and multi-modal Phi-2. Our key findings show that these MLLMs rely on MLP and self-attention blocks in much earlier layers for information storage, compared to LLMs whose mid-layer MLPs are more important. We also show that a consistent small subset of visual tokens output by the vision encoder are responsible for transferring information from the image to these causal blocks. We validate these mechanisms by introducing MultEdit, a model-editing algorithm that can correct errors and insert new long-tailed information into MLLMs by targeting these causal blocks.
Localization plays a crucial role in enhancing the practicality and precision of VQA systems. By enabling fine-grained identification and interaction with specific parts of an object, it significantly improves the system's ability to provide contextually relevant and spatially accurate responses, crucial for applications in dynamic environments like robotics and augmented reality. However, traditional systems face challenges in accurately mapping objects within images to generate nuanced and spatially aware responses. In this work, we introduce "Detect2Interact", which addresses these challenges by introducing an advanced approach for fine-grained object visual key field detection. First, we use the segment anything model (SAM) to generate detailed spatial maps of objects in images. Next, we use Vision Studio to extract semantic object descriptions. Third, we employ GPT-4's common sense knowledge, bridging the gap between an object's semantics and its spatial map. As a result, Detect2Interact achieves consistent qualitative results on object key field detection across extensive test cases and outperforms the existing VQA system with object detection by providing a more reasonable and finer visual representation.
The ability to perform complex reasoning across multimodal inputs is essential for models to effectively interact with humans in real-world scenarios. Advancements in vision-language models have significantly improved performance on tasks that require processing explicit and direct textual inputs, such as Visual Question Answering (VQA) and Visual Grounding (VG). However, less attention has been given to improving the model capabilities to comprehend nuanced and ambiguous forms of communication. This presents a critical challenge, as human language in real-world interactions often convey hidden intentions that rely on context for accurate interpretation. To address this gap, we propose VAGUE, a multimodal benchmark comprising 3.9K indirect human utterances paired with corresponding scenes. Additionally, we contribute a model-based pipeline for generating prompt-solution pairs from input images. Our work aims to delve deeper into the ability of models to understand indirect communication and seek to contribute to the development of models capable of more refined and human-like interactions. Extensive evaluation on multiple VLMs reveals that mainstream models still struggle with indirect communication when required to perform complex linguistic and visual reasoning. We release our code and data at https://github.com/Hazel-Heejeong-Nam/VAGUE.git.
대규모 멀티모달 모델의 최근 발전은 비전-언어 과제에서 눈부신 성과를 이끌었지만, 풍부하고 명시적으로 정렬된 청각 단서를 포함한 데이터셋의 부족으로 인해 포괄적인 옴니모달 이해는 여전히 충분히 탐구되지 않은 상태이다. 이 격차를 해소하기 위해, 우리는 시각 및 청각 의미론을 분리하도록 설계된 대규모 데이터셋인 AVDC(Audio-Visual Decoupled Captions)를 제시한다. 구체적으로, 기성 모델을 활용하여 영상에 세 가지 캡션—시각 전용(V), 오디오 전용(A), 오디오-시각 결합(AV)—을 자동으로 주석 처리하는 파이프라인을 제안한다. 이 분리된 구조는 각 모달리티 고유의 미묘한 특성과 복잡한 교차 모달 상호작용을 명시적으로 포착한다. 이를 바탕으로, 오디오-시각 추론을 촉진하기 위한 사고의 연쇄(Chain-of-Thought) 기반 질의응답 데이터셋인 AVDC-QA-CoT를 소개한다. 이러한 자원을 충분히 활용하기 위해, AVDC에 대한 옴니모달 캡션 생성 사전학습에 이어 AVDC-QA-CoT에 대한 지시 미세조정을 수행하는 2단계 학습 패러다임을 채택한다. 영상 캡션 생성, 오디오 중심 분석, 옴니모달 벤치마크를 아우르는 다양한 다운스트림 과제에 걸친 광범위한 실험을 통해 일관되고 유의미한 성능 향상이 확인되었으며, 이는 제안된 데이터셋과 학습 전략이 옴니모달 인식 발전에 기여함을 보여준다. 코드와 데이터셋은 https://radiant0726.github.io/AVDC-web/ 에서 공개된다.
While recent Multimodal Large Language Models exhibit impressive capabilities for general multimodal tasks, specialized domains like music necessitate tailored approaches. Music Audio-Visual Question Answering (Music AVQA) particularly underscores this, presenting unique challenges with its continuous, densely layered audio-visual content, intricate temporal dynamics, and the critical need for domain-specific knowledge. Through a systematic analysis of Music AVQA datasets and methods, this position paper identifies that specialized input processing, architectures incorporating dedicated spatial-temporal designs, and music-specific modeling strategies are critical for success in this domain. Our study provides valuable insights for researchers by highlighting effective design patterns empirically linked to strong performance, proposing concrete future directions for incorporating musical priors, and aiming to establish a robust foundation for advancing multimodal musical understanding. This work is intended to inspire broader attention and further research, supported by a continuously updated anonymous GitHub repository of relevant papers: https://github.com/xid32/Survey4MusicAVQA.
비전-언어 모델(VLM)은 긴 비디오 이해에서 상당한 진전을 이루었으며, 표준 백본 모델은 일반적으로 전체 비디오에서 샘플링된 프레임으로부터 질문에 답합니다. 그러나 비디오가 길어질수록 전체 비디오 문맥에는 필연적으로 질문과 무관한 시간적 내용이 더 많이 포함되며, 이는 특정 질문에 답하는 데 필요한 증거로부터 모델의 주의를 분산시킬 수 있습니다. 본 연구에서는 질문 관련 증거를 담고 있는 짧은 주석 달린 단서 구간(clue interval)에 시각적 입력을 집중시키는 것이, 해당 전체 비디오를 사용하는 것과 비교해 더 적은 입력 프레임을 요구하면서도 모델 규모 전반에 걸쳐 예측 정확도를 일관되게 향상시킨다는 점을 실증적으로 발견했습니다. 이러한 발견을 바탕으로, 긴 비디오 이해를 위한 단서 특권 기반 온-정책 자기 증류 프레임워크인 Clue-OPSD를 제안합니다. 학습(training) 중에는 전체 비디오를 입력받는 학생 모델이 학생이 생성한 궤적을 따라 다음 토큰(token) 분포를 정렬함으로써, 해당 단서 구간에 조건화된 자기 교사 모델로부터 배웁니다. 따라서 Clue-OPSD는 정답 라벨에 의존하지 않고 단서 구간을 특권적 감독 신호로 활용하는 한편, 추론(inference) 시점에는 단서 주석이나 추가 모듈을 전혀 필요로 하지 않습니다. 여러 긴 비디오 이해 벤치마크 및 Qwen3.5 모델 규모에 걸친 광범위한 실험을 통해, 해당 백본 모델 대비 일관된 성능 향상과 지도 사후 학습(post-training) 베이스라인 대비 강력한 성능을 입증합니다.
We introduce a novel framework for evaluating multimodal deep learning models with respect to their language understanding and generalization abilities. In this approach, artificial data is automatically generated according to the experimenter's specifications. The content of the data, both during training and evaluation, can be controlled in detail, which enables tasks to be created that require true generalization abilities, in particular the combination of previously introduced concepts in novel ways. We demonstrate the potential of our methodology by evaluating various visual question answering models on four different tasks, and show how our framework gives us detailed insights into their capabilities and limitations. By open-sourcing our framework, we hope to stimulate progress in the field of multimodal language understanding.
Multimodal models leverage large-scale pre-training to achieve strong but still imperfect performance on tasks such as image captioning, visual question answering, and cross-modal retrieval. In this paper, we present a simple and efficient method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization (NNN). We show an improvement on retrieval metrics in both text retrieval and image retrieval for all of the contrastive models that we tested (CLIP, BLIP, ALBEF, SigLIP, BEiT) and for both of the datasets that we used (MS-COCO and Flickr30k). NNN requires a reference database, but does not require any training on this database, and can even increase the retrieval accuracy of a model after finetuning.
Remote sensing visual question answering (RSVQA) has been involved in several research in recent years, leading to an increase in new methods. RSVQA automatically extracts information from satellite images, so far only optical, and a question to automatically search for the answer in the image and provide it in a textual form. In our research, we study whether Synthetic Aperture Radar (SAR) images can be beneficial to this field. We divide our study into three phases which include classification methods and VQA. In the first one, we explore the classification results of SAR alone and investigate the best method to extract information from SAR data. Then, we study the combination of SAR and optical data. In the last phase, we investigate how SAR images and a combination of different modalities behave in RSVQA compared to a method only using optical images. We conclude that adding the SAR modality leads to improved performances, although further research on using SAR data to automatically answer questions is needed as well as more balanced datasets.
Humans have the innate capability to answer diverse questions, which is rooted in the natural ability to correlate different concepts based on their semantic relationships and decompose difficult problems into sub-tasks. On the contrary, existing visual reasoning methods assume training samples that capture every possible object and reasoning problem, and rely on black-boxed models that commonly exploit statistical priors. They have yet to develop the capability to address novel objects or spurious biases in real-world scenarios, and also fall short of interpreting the rationales behind their decisions. Inspired by humans' reasoning of the visual world, we tackle the aforementioned challenges from a compositional perspective, and propose an integral framework consisting of a principled object factorization method and a novel neural module network. Our factorization method decomposes objects based on their key characteristics, and automatically derives prototypes that represent a wide range of objects. With these prototypes encoding important semantics, the proposed network then correlates objects by measuring their similarity on a common semantic space and makes decisions with a compositional reasoning process. It is capable of answering questions with diverse objects regardless of their availability during training, and overcoming the issues of biased question-answer distributions. In addition to the enhanced generalizability, our framework also provides an interpretable interface for understanding the decision-making process of models. Our code is available at https://github.com/szzexpoi/POEM.
비전-언어 모델(VLM)은 이미지 캡셔닝(image captioning)과 시각적 질의응답(VQA)과 같이 이미지와 텍스트를 공동으로 이해해야 하는 과제에서 인상적인 성능을 보여 왔지만, 내부 처리 과정에 대한 이해는 여전히 제한적이다. 최근 희소 오토인코더(SAE)는 VLM에 인코딩된 개념의 해석을 지원하는 유망한 도구로 부상했다. 그러나 대부분의 SAE 기반 접근법은 다중모달 개념을 무시하고 텍스트 또는 시각 개념만을 각각 따로 다룬다. 이러한 한계는 두 모달리티를 통합하는 개념이 잘못 분류될 수 있기 때문에 VLM에 대한 포괄적 이해를 방해한다. 또한 이전의 시각적 접근법은 종종 모호하거나 불완전한 저품질 시각 개념 설명을 생성하여, 모델 추론을 이해하는 데 있어 유용성을 제한한다. 우리는 VLM에서 시각, 텍스트, 다중모달 개념을 추출하고 분석하기 위한 SAE 기반 프레임워크를 제안한다. 각 뉴런에 대해 사람이 해석 가능한 후보 개념을 제안하고, 코사인 유사도 점수를 사용하여 해당 개념과 데이터셋 샘플 간의 정합을 계산한다. VQA 데이터셋(LLaVA-NeXT)에서의 실험은 우리 프레임워크가 기존 SAE 기반 방법 대비 시각 개념 품질을 최대 45%까지 향상시키면서, 높은 텍스트 개념 품질을 유지하고 다중모달 개념의 체계적 식별을 가능하게 함을 보여준다. 본 연구는 VLM의 개념 공간에 대한 새로운 통찰을 제공하며, 시각, 텍스트, 다중모달 개념을 구별하는 구조화된 접근법을 제시한다. 코드는 https://github.com/PHDLanza/Multidata_SAE 에서 이용할 수 있다.
Building a foundation model for 3D vision is a complex challenge that remains unsolved. Towards that goal, it is important to understand the 3D reasoning capabilities of current models as well as identify the gaps between these models and humans. Therefore, we construct a new 3D visual understanding benchmark named UniQA-3D. UniQA-3D covers fundamental 3D vision tasks in the Visual Question Answering (VQA) format. We evaluate state-of-the-art Vision-Language Models (VLMs), specialized models, and human subjects on it. Our results show that VLMs generally perform poorly, while the specialized models are accurate but not robust, failing under geometric perturbations. In contrast, human vision continues to be the most reliable 3D visual system. We further demonstrate that neural networks align more closely with human 3D vision mechanisms compared to classical computer vision methods, and Transformer-based networks such as ViT align more closely with human 3D vision mechanisms than CNNs. We hope our study will benefit the future development of foundation models for 3D vision. Code is available at https://github.com/princeton-vl/UniQA-3D .
Large Multimodal Models (LMMs) have demonstrated impressive capabilities in visual-language tasks but face significant deployment challenges due to their high computational demands. While recent token reduction methods show promise for accelerating LMMs, they typically require extensive retraining or fine-tuning, making them impractical for many state-of-the-art models, especially those with proprietary training data. We propose freePruner, a training-free token reduction approach that can be directly applied to any open-source LMM without additional training. Unlike existing methods that rely heavily on token merging operations, freePruner employs a two-stage token selection strategy: (1) identifying pivotal tokens that capture high-level semantic information using our designed contribution degree metric, and (2) selecting complementary tokens that preserve essential low-level visual details through attention pattern analysis. Extensive experiments demonstrate that freePruner achieves 2x acceleration while maintaining comparable performance across mainstream visual question-answering benchmarks in the training-free setting. Moreover, freePruner is orthogonal to and can be combined with other post-training acceleration techniques, such as post-training quantization, providing a practical solution for efficient LMM deployment.
Despite recent advances on multi-modal models, 3D spatial reasoning remains a challenging task for state-of-the-art open-source and proprietary models. Recent studies explore data-driven approaches and achieve enhanced spatial reasoning performance by fine-tuning models on 3D-related visual question-answering data. However, these methods typically perform spatial reasoning in an implicit manner and often fail on questions that are trivial to humans, even with long chain-of-thought reasoning. In this work, we introduce SpatialReasoner, a novel large vision-language model (LVLM) that addresses 3D spatial reasoning with explicit 3D representations shared between multiple stages--3D perception, computation, and reasoning. Explicit 3D representations provide a coherent interface that supports advanced 3D spatial reasoning and improves the generalization ability to novel question types. Furthermore, by analyzing the explicit 3D representations in multi-step reasoning traces of SpatialReasoner, we study the factual errors and identify key shortcomings of current LVLMs. Results show that our SpatialReasoner achieves improved performance on a variety of spatial reasoning benchmarks, outperforming Gemini 2.0 by 9.2% on 3DSRBench, and generalizes better when evaluating on novel 3D spatial reasoning questions. Our study bridges the 3D parsing capabilities of prior visual foundation models with the powerful reasoning abilities of large language models, opening new directions for 3D spatial reasoning.
Large multimodal models (LMMs) have achieved impressive progress in vision-language understanding, yet they face limitations in real-world applications requiring complex reasoning over a large number of images. Existing benchmarks for multi-image question-answering are limited in scope, each question is paired with only up to 30 images, which does not fully capture the demands of large-scale retrieval tasks encountered in the real-world usages. To reduce these gaps, we introduce two document haystack benchmarks, dubbed DocHaystack and InfoHaystack, designed to evaluate LMM performance on large-scale visual document retrieval and understanding. Additionally, we propose V-RAG, a novel, vision-centric retrieval-augmented generation (RAG) framework that leverages a suite of multimodal vision encoders, each optimized for specific strengths, and a dedicated question-document relevance module. V-RAG sets a new standard, with a 9% and 11% improvement in Recall@1 on the challenging DocHaystack-1000 and InfoHaystack-1000 benchmarks, respectively, compared to the previous best baseline models. Additionally, integrating V-RAG with LMMs enables them to efficiently operate across thousands of images, yielding significant improvements on our DocHaystack and InfoHaystack benchmarks. Our code and datasets are available at https://github.com/Vision-CAIR/dochaystacks
It is still challenging to build an AI system that can perform tasks that involve vision and language at human level. So far, researchers have singled out individual tasks separately, for each of which they have designed networks and trained them on its dedicated datasets. Although this approach has seen a certain degree of success, it comes with difficulties of understanding relations among different tasks and transferring the knowledge learned for a task to others. We propose a multi-task learning approach that enables to learn vision-language representation that is shared by many tasks from their diverse datasets. The representation is hierarchical, and prediction for each task is computed from the representation at its corresponding level of the hierarchy. We show through experiments that our method consistently outperforms previous single-task-learning methods on image caption retrieval, visual question answering, and visual grounding. We also analyze the learned hierarchical representation by visualizing attention maps generated in our network.
Scene graphs have become an important form of structured knowledge for tasks such as for image generation, visual relation detection, visual question answering, and image retrieval. While visualizing and interpreting word embeddings is well understood, scene graph embeddings have not been fully explored. In this work, we train scene graph embeddings in a layout generation task with different forms of supervision, specifically introducing triplet super-vision and data augmentation. We see a significant performance increase in both metrics that measure the goodness of layout prediction, mean intersection-over-union (mIoU)(52.3% vs. 49.2%) and relation score (61.7% vs. 54.1%),after the addition of triplet supervision and data augmentation. To understand how these different methods affect the scene graph representation, we apply several new visualization and evaluation methods to explore the evolution of the scene graph embedding. We find that triplet supervision significantly improves the embedding separability, which is highly correlated with the performance of the layout prediction model.