PyTorchKR Papers

“Visual Question Answering”

총 2,804편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2611.06434v1cs.AI

EviSeek: 임바디드 능동 정보 탐색을 위한 증거 중심 에이전트 프레임워크

EviSeek: An Evidence-Centric Agentic Framework for Embodied Active Information Seeking

능동적 임바디드 질의응답은 에이전트가 시점 변경과 물리적 상호작용을 통해 증거를 획득할 것을 요구한다. 핵심 과제는 방향성 없는 상호작용, 즉 질문에 답하기 위해 어떤 증거를 드러내야 하는지 specifying하지 않은 채 그럴듯한 행동을 취하는 일을 피하는 것이다. 행동 계획과 누적된 이력만으로는 상호작용의 정보적 목적이 암묵적인 채로 남을 수 있다. 본 연구는 질문에서 도출된 증거 요구를 명시적인 획득 목표로 만들어, 관측에 조건화된 행동과 선택적으로 유지된 증거를 에이전트가 찾는 정보와 연결하는 학습 불필요 프레임워크 EviSeek을 제안한다. ESI-Bench의 3,436개 질문에서 EviSeek은 Single-model 에이전트 대비 정확도를 Qwen3-VL-32B에서 7.1%p, Gemini 3.1 Pro에서 3.7%p 향상시키며, 시각적 근거 가중 정확도를 개선하고 평균 토큰 사용량을 줄인다. 이러한 결과는 에이전트가 질문에 답하기 위해 무엇을 알아야 하는지를 중심으로 행동 선택과 증거 유지를 구성하여 상호작용의 정보적 목적을 명시하는 접근을 뒷받침한다.

arXiv:2610.01180v1cs.CV

스켈레톤 및 전략 프롬프팅: 비전-언어 모델을 위한 학습 불필요 부정 이해

Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models

다양한 시각적 질의응답(VQA) 태스크에서 비전-언어 모델(VLM)이 뛰어난 성능을 보임에도 불구하고, 이러한 모델들은 지속적으로 부정(negation)을 이해하는 데 어려움을 겪으며 부정절이 포함된 질문에 대해 잘못된 답변을 생성합니다. 이러한 한계를 해결하기 위해, 본 논문에서는 매개변수 업데이트 없이 VLM의 부정 이해 능력을 향상시키는 학습 불필요(training-free) 문맥 내 학습(ICL) 방식인 스켈레톤 및 전략 프롬프팅(Skeleton-and-Strategy Prompting, SSP)을 제안합니다. 부정 질문이 주어지면, 제안하는 방법은 먼저 질문의 기저 구조를 스켈레톤으로 추상화하고, 경량화된 질문 풀에서 동일한 스켈레톤을 가진 소수의 질문 집합을 검색한 다음, VLM이 이들의 공유된 부정 패턴을 분석하여 단일 문장으로 된 답변 전략을 합성하도록 프롬프트합니다. 이렇게 생성된 스켈레톤과 전략은 테스트 샘플 앞에 추가되어 모델이 부정 문제를 올바르게 해결하도록 안내합니다. 여러 부정 VQA 벤치마크에 대한 실험을 통해 SSP가 연산 효율성을 유지하면서도 부정 중심 VQA 태스크에서 최첨단(SOTA) 성능을 달성함을 보입니다.

arXiv:2112.06343공식 코드 · Papers with Code

Change Detection Meets Visual Question Answering

The Earth's surface is continually changing, and identifying changes plays an important role in urban planning and sustainability. Although change detection techniques have been successfully developed for many years, these techniques are still limited to experts and facilitators in related fields. In order to provide every user with flexible access to change information and help them better understand land-cover changes, we introduce a novel task: change detection-based visual question answering (CDVQA) on multi-temporal aerial images. In particular, multi-temporal images can be queried to obtain high level change-based information according to content changes between two input images. We first build a CDVQA dataset including multi-temporal image-question-answer triplets using an automatic question-answer generation method. Then, a baseline CDVQA framework is devised in this work, and it contains four parts: multi-temporal feature encoding, multi-temporal fusion, multi-modal fusion, and answer prediction. In addition, we also introduce a change enhancing module to multi-temporal feature encoding, aiming at incorporating more change-related information. Finally, effects of different backbones and multi-temporal fusion strategies are studied on the performance of CDVQA task. The experimental results provide useful insights for developing better CDVQA models, which are important for future research on this task.

arXiv:2403.03864공식 코드 · Papers with Code

Are Language Models Puzzle Prodigies? Algorithmic Puzzles Unveil Serious Challenges in Multimodal Reasoning

This paper introduces the novel task of multimodal puzzle solving, framed within the context of visual question-answering. We present a new dataset, AlgoPuzzleVQA designed to challenge and evaluate the capabilities of multimodal language models in solving algorithmic puzzles that necessitate both visual understanding, language understanding, and complex algorithmic reasoning. We create the puzzles to encompass a diverse array of mathematical and algorithmic topics such as boolean logic, combinatorics, graph theory, optimization, search, etc., aiming to evaluate the gap between visual data interpretation and algorithmic problem-solving skills. The dataset is generated automatically from code authored by humans. All our puzzles have exact solutions that can be found from the algorithm without tedious human calculations. It ensures that our dataset can be scaled up arbitrarily in terms of reasoning complexity and dataset size. Our investigation reveals that large language models (LLMs) such as GPT4V and Gemini exhibit limited performance in puzzle-solving tasks. We find that their performance is near random in a multi-choice question-answering setup for a significant number of puzzles. The findings emphasize the challenges of integrating visual, language, and algorithmic knowledge for solving complex reasoning problems.

arXiv:2006.15631

Improving VQA and its Explanations \\ by Comparing Competing Explanations

Most recent state-of-the-art Visual Question Answering (VQA) systems are opaque black boxes that are only trained to fit the answer distribution given the question and visual content. As a result, these systems frequently take shortcuts, focusing on simple visual concepts or question priors. This phenomenon becomes more problematic as the questions become complex that requires more reasoning and commonsense knowledge. To address this issue, we present a novel framework that uses explanations for competing answers to help VQA systems select the correct answer. By training on human textual explanations, our framework builds better representations for the questions and visual content, and then reweights confidences in the answer candidates using either generated or retrieved explanations from the training set. We evaluate our framework on the VQA-X dataset, which has more difficult questions with human explanations, achieving new state-of-the-art results on both VQA and its explanations.

arXiv:2502.11747

Multi-Modal Retrieval Augmentation for Open-Ended and Knowledge-Intensive Video Question Answering

While current video question answering systems perform well on some tasks requiring only direct visual understanding, they struggle with questions demanding knowledge beyond what is immediately observable in the video content. We refer to this challenging scenario as knowledge-intensive video question answering (KI-VideoQA), where models must retrieve and integrate external information with visual understanding to generate accurate responses. This work presents the first attempt to (1) study multi-modal retrieval-augmented generation for KI-VideoQA, and (2) go beyond multi-choice questions by studying open-ended questions in this task. Through an extensive empirical study of state-of-the-art retrieval and vision language models in both zero-shot and fine-tuned settings, we explore how different retrieval augmentation strategies can enhance knowledge integration in KI-VideoQA. We analyze three key aspects: (1) model's effectiveness across different information sources and modalities, (2) the impact of heterogeneous multi-modal context integration, and (3) model's effectiveness across different query formulation and retrieval result consumption. Our results suggest that while retrieval augmentation generally improves performance, its effectiveness varies significantly based on modality choice and retrieval strategy. Additionally, we find that successful knowledge integration often requires careful consideration of query formulation and optimal retrieval depth. Our exploration advances state-of-the-art accuracy for multiple choice questions by over 17.5% on the KnowIT VQA dataset.

arXiv:2303.00534공식 코드 · Papers with Code

RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training

Vision-and-language multi-modal pretraining and fine-tuning have shown great success in visual question answering (VQA). Compared to general domain VQA, the performance of biomedical VQA suffers from limited data. In this paper, we propose a retrieval-augmented pretrain-and-finetune paradigm named RAMM for biomedical VQA to overcome the data limitation issue. Specifically, we collect a new biomedical dataset named PMCPM which offers patient-based image-text pairs containing diverse patient situations from PubMed. Then, we pretrain the biomedical multi-modal model to learn visual and textual representation for image-text pairs and align these representations with image-text contrastive objective (ITC). Finally, we propose a retrieval-augmented method to better use the limited data. We propose to retrieve similar image-text pairs based on ITC from pretraining datasets and introduce a novel retrieval-attention module to fuse the representation of the image and the question with the retrieved images and texts. Experiments demonstrate that our retrieval-augmented pretrain-and-finetune paradigm obtains state-of-the-art performance on Med-VQA2019, Med-VQA2021, VQARAD, and SLAKE datasets. Further analysis shows that the proposed RAMM and PMCPM can enhance biomedical VQA performance compared with previous resources and methods. We will open-source our dataset, codes, and pretrained model.

arXiv:2512.14926v2cs.CL

루마니아어 비전-언어 모델을 위한 매개변수 효율적 멀티모달 지시 미세조정

Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models

저자원 언어에 집중하는 것은 생성 AI의 민주화를 향한 필수적인 단계입니다. 본 연구에서는 루마니아어의 멀티모달 자연어 처리(NLP) 자원 격차를 줄이는 데 기여합니다. 널리 알려진 Flickr30K 데이터셋을 루마니아어로 번역하고, 오픈소스 대규모 언어 모델(LLM)을 활용하여 시각적 질의응답(visual question answering)을 위해 이를 추가로 확장합니다. 오픈소스 비전-언어 모델(VLM)을 루마니아어 시각적 질의응답에 대해 미세조정함으로써 데이터셋의 유용성을 입증합니다. 세 가지 널리 사용되는 모델 패밀리인 LLaMA 3.2, LLaVA 1.6, Qwen2에서 VLM을 선택하며, 미세조정에는 매개변수 효율적인 LoRA 기법을 적용합니다. 우리의 모델은 시각적 QA에서 루마니아어 능력이 향상되었을 뿐만 아니라, 루마니아어 이미지 설명 생성과 같이 학습되지 않은 태스크에서도 성능 개선을 보입니다. 70억 매개변수 모델인 Qwen2-VL-RoVQA는 두 태스크 모두에서 최고 점수를 기록하며, 원본 버전 대비 VQA와 캡셔닝에서 각각 BERTScore F1 기준 +2.29% 및 +4.45%의 향상을 달성합니다. 마지막으로, 모델들은 원본 형태 대비 문법 오류가 크게 감소하여, 언어 이해뿐만 아니라 루마니아어 유창성에서도 향상이 이루어졌음을 나타냅니다.

arXiv:2412.19304

Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries

Video Question Answering (Video QA) is a challenging video understanding task that requires models to comprehend entire videos, identify the most relevant information based on contextual cues from a given question, and reason accurately to provide answers. Recent advancements in Multimodal Large Language Models (MLLMs) have transformed video QA by leveraging their exceptional commonsense reasoning capabilities. This progress is largely driven by the effective alignment between visual data and the language space of MLLMs. However, for video QA, an additional space-time alignment poses a considerable challenge for extracting question-relevant information across frames. In this work, we investigate diverse temporal modeling techniques to integrate with MLLMs, aiming to achieve question-guided temporal modeling that leverages pre-trained visual and textual alignment in MLLMs. We propose T-Former, a novel temporal modeling method that creates a question-guided temporal bridge between frame-wise visual perception and the reasoning capabilities of LLMs. Our evaluation across multiple video QA benchmarks demonstrates that T-Former competes favorably with existing temporal modeling approaches and aligns with recent advancements in video QA.

arXiv:2503.21747

CTRL-O: Language-Controllable Object-Centric Visual Representation Learning

Object-centric representation learning aims to decompose visual scenes into fixed-size vectors called "slots" or "object files", where each slot captures a distinct object. Current state-of-the-art object-centric models have shown remarkable success in object discovery in diverse domains, including complex real-world scenes. However, these models suffer from a key limitation: they lack controllability. Specifically, current object-centric models learn representations based on their preconceived understanding of objects, without allowing user input to guide which objects are represented. Introducing controllability into object-centric models could unlock a range of useful capabilities, such as the ability to extract instance-specific representations from a scene. In this work, we propose a novel approach for user-directed control over slot representations by conditioning slots on language descriptions. The proposed ConTRoLlable Object-centric representation learning approach, which we term CTRL-O, achieves targeted object-language binding in complex real-world scenes without requiring mask supervision. Next, we apply these controllable slot representations on two downstream vision language tasks: text-to-image generation and visual question answering. The proposed approach enables instance-specific text-to-image generation and also achieves strong performance on visual question answering.

arXiv:2111.01026공식 코드 · Papers with Code

Introspective Distillation for Robust Question Answering

Question answering (QA) models are well-known to exploit data bias, e.g., the language prior in visual QA and the position bias in reading comprehension. Recent debiasing methods achieve good out-of-distribution (OOD) generalizability with a considerable sacrifice of the in-distribution (ID) performance. Therefore, they are only applicable in domains where the test distribution is known in advance. In this paper, we present a novel debiasing method called Introspective Distillation (IntroD) to make the best of both worlds for QA. Our key technical contribution is to blend the inductive bias of OOD and ID by introspecting whether a training sample fits in the factual ID world or the counterfactual OOD one. Experiments on visual QA datasets VQA v2, VQA-CP, and reading comprehension dataset SQuAD demonstrate that our proposed IntroD maintains the competitive OOD performance compared to other debiasing methods, while sacrificing little or even achieving better ID performance compared to the non-debiasing ones.

arXiv:2405.18831

Evaluating Zero-Shot GPT-4V Performance on 3D Visual Question Answering Benchmarks

As interest in "reformulating" the 3D Visual Question Answering (VQA) problem in the context of foundation models grows, it is imperative to assess how these new paradigms influence existing closed-vocabulary datasets. In this case study, we evaluate the zero-shot performance of foundational models (GPT-4 Vision and GPT-4) on well-established 3D VQA benchmarks, namely 3D-VQA and ScanQA. We provide an investigation to contextualize the performance of GPT-based agents relative to traditional modeling approaches. We find that GPT-based agents without any fine-tuning perform on par with the closed vocabulary approaches. Our findings corroborate recent results that "blind" models establish a surprisingly strong baseline in closed-vocabulary settings. We demonstrate that agents benefit significantly from scene-specific vocabulary via in-context textual grounding. By presenting a preliminary comparison with previous baselines, we hope to inform the community's ongoing efforts to refine multi-modal 3D benchmarks.

arXiv:2104.06039

MultiModalQA: Complex Question Answering over Text, Tables and Images

When answering complex questions, people can seamlessly combine information from visual, textual and tabular sources. While interest in models that reason over multiple pieces of evidence has surged in recent years, there has been relatively little work on question answering models that reason across multiple modalities. In this paper, we present MultiModalQA(MMQA): a challenging question answering dataset that requires joint reasoning over text, tables and images. We create MMQA using a new framework for generating complex multi-modal questions at scale, harvesting tables from Wikipedia, and attaching images and text paragraphs using entities that appear in each table. We then define a formal language that allows us to take questions that can be answered from a single modality, and combine them to generate cross-modal questions. Last, crowdsourcing workers take these automatically-generated questions and rephrase them into more fluent language. We create 29,918 questions through this procedure, and empirically demonstrate the necessity of a multi-modal multi-hop approach to solve our task: our multi-hop model, ImplicitDecomp, achieves an average F1of 51.7 over cross-modal questions, substantially outperforming a strong baseline that achieves 38.2 F1, but still lags significantly behind human performance, which is at 90.1 F1

arXiv:2104.05832공식 코드 · Papers with Code

SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning

This paper proposes a question-answering (QA) benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior work and is challenging for state-of-the-art language models (LM). We propose a distant supervision method to improve on this task. Specifically, we design grammar and reasoning rules to automatically generate a spatial description of visual scenes and corresponding QA pairs. Experiments show that further pretraining LMs on these automatically generated data significantly improves LMs' capability on spatial understanding, which in turn helps to better solve two external datasets, bAbI, and boolQ. We hope that this work can foster investigations into more sophisticated models for spatial reasoning over text.

arXiv:2502.16636공식 코드 · Papers with Code

Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries

Retrieval-Augmented Generation (RAG) is a popular approach for enhancing Large Language Models (LLMs) by addressing their limitations in verifying facts and answering knowledge-intensive questions. As the research in LLM extends their capability to handle input modality other than text, e.g. image, several multimodal RAG benchmarks are proposed. Nonetheless, they mainly use textual knowledge bases as the primary source of evidences for augmentation. There still lack benchmarks designed to evaluate images as augmentation in RAG systems and how they leverage visual knowledge. We propose Visual-RAG, a novel Question Answering benchmark that emphasizes visual knowledge intensive questions. Unlike prior works relying on text-based evidence, Visual-RAG necessitates text-to-image retrieval and integration of relevant clue images to extract visual knowledge as evidence. With Visual-RAG, we evaluate 5 open-sourced and 3 proprietary Multimodal LLMs (MLLMs), revealing that images can serve as good evidence in RAG; however, even the SoTA models struggle with effectively extracting and utilizing visual knowledge

arXiv:2102.02779공식 코드 · Papers with Code

Unifying Vision-and-Language Tasks via Text Generation

Existing methods for vision-and-language learning typically require designing task-specific architectures and objectives for each task. For example, a multi-label answer classifier for visual question answering, a region scorer for referring expression comprehension, and a language decoder for image captioning, etc. To alleviate these hassles, in this work, we propose a unified framework that learns different tasks in a single architecture with the same language modeling objective, i.e., multimodal conditional text generation, where our models learn to generate labels in text based on the visual and textual inputs. On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-language models. Moreover, our generative approach shows better generalization ability on questions that have rare answers. Also, we show that our framework allows multi-task learning in a single architecture with a single set of parameters, achieving similar performance to separately optimized single-task models. Our code is publicly available at: https://github.com/j-min/VL-T5

arXiv:2206.12533

From Shallow to Deep: Compositional Reasoning over Graphs for Visual Question Answering

In order to achieve a general visual question answering (VQA) system, it is essential to learn to answer deeper questions that require compositional reasoning on the image and external knowledge. Meanwhile, the reasoning process should be explicit and explainable to understand the working mechanism of the model. It is effortless for human but challenging for machines. In this paper, we propose a Hierarchical Graph Neural Module Network (HGNMN) that reasons over multi-layer graphs with neural modules to address the above issues. Specifically, we first encode the image by multi-layer graphs from the visual, semantic and commonsense views since the clues that support the answer may exist in different modalities. Our model consists of several well-designed neural modules that perform specific functions over graphs, which can be used to conduct multi-step reasoning within and between different graphs. Compared to existing modular networks, we extend visual reasoning from one graph to more graphs. We can explicitly trace the reasoning process according to module weights and graph attentions. Experiments show that our model not only achieves state-of-the-art performance on the CRIC dataset but also obtains explicit and explainable reasoning procedures.

arXiv:2609.02486v3cs.IR

ViSAR: 문서 시각 질의응답을 위한 학습이 필요 없는 적응형 $k$ 검색

ViSAR: Training-Free Adaptive-$k$ Retrieval for Document Visual Question Answering

문서 시각 질의응답(DocVQA)은 검색 증강 생성(RAG)을 활용하는 경우가 많으며, 여기서는 대규모 시각-언어 모델(LVLM)이 답변을 생성하기 전에 사용자 질의와 관련된 문서 페이지를 식별하는 데 후기 상호작용 인코더가 흔히 사용된다. 기존 접근법은 일반적으로 질의의 복잡도와 무관하게 고정된 top-$k$ 개수의 페이지를 검색하는데, 이는 LVLM의 지연 시간을 늘리고 더 적은 페이지로 충분한 경우 답변 정확도를 떨어뜨릴 수 있다. 우리는 후기 상호작용 시각 문서 검색을 위한, 학습이 필요 없고 예산을 인지하는 적응형 $k$ 검색 방법인 ViSAR(Visual Semantic Activation Retrieval)를 제안한다. ViSAR는 임베딩 공간에서 직접 동작하여 질의와 관련된 의미를 강조하는 질의 조건부 페이지 수준 유사도 행렬을 구성하고, 주어진 페이지 예산 내에서 검색할 페이지 수를 동적으로 결정한다. 우리는 여러 데이터셋, 인코더, LVLM에 걸쳐 본 방법을 평가한다. ViSAR는 간결하고 질의에 적응된 페이지 집합을 검색하여, 고정 top-$k$ 및 적응형 검색 휴리스틱과 비교해 통계적으로 유의한 정확도 손실 없이, 그리고 여러 설정에서는 유의한 향상과 함께, 종단 간 RAG 지연 시간을 최대 58.9\%까지 줄인다. 더 나아가 유사도 행렬의 구조가 답변 정확도와 상관관계가 있음을 보이며, 이는 검색 품질을 인지하는 문서 이해라는 향후 방향을 시사한다.

arXiv:2104.14741

Chop Chop BERT: Visual Question Answering by Chopping VisualBERT's Heads

Vision-and-Language (VL) pre-training has shown great potential on many related downstream tasks, such as Visual Question Answering (VQA), one of the most popular problems in the VL field. All of these pre-trained models (such as VisualBERT, ViLBERT, LXMERT and UNITER) are built with Transformer, which extends the classical attention mechanism to multiple layers and heads. To investigate why and how these models work on VQA so well, in this paper we explore the roles of individual heads and layers in Transformer models when handling $12$ different types of questions. Specifically, we manually remove (chop) heads (or layers) from a pre-trained VisualBERT model at a time, and test it on different levels of questions to record its performance. As shown in the interesting echelon shape of the result matrices, experiments reveal different heads and layers are responsible for different question types, with higher-level layers activated by higher-level visual reasoning questions. Based on this observation, we design a dynamic chopping module that can automatically remove heads and layers of the VisualBERT at an instance level when dealing with different questions. Our dynamic chopping module can effectively reduce the parameters of the original model by 50%, while only damaging the accuracy by less than 1% on the VQA task.

arXiv:2505.15517

Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets

Vision-Language Models (VLMs) acquire real-world knowledge and general reasoning ability through Internet-scale image-text corpora. They can augment robotic systems with scene understanding and task planning, and assist visuomotor policies that are trained on robot trajectory data. We explore the reverse paradigm - using rich, real, multi-modal robot trajectory data to enhance and evaluate VLMs. In this paper, we present Robo2VLM, a Visual Question Answering (VQA) dataset generation framework for VLMs. Given a human tele-operated robot trajectory, Robo2VLM derives ground-truth from non-visual and non-descriptive sensory modalities, such as end-effector pose, gripper aperture, and force sensing. Based on these modalities, it segments the robot trajectory into a sequence of manipulation phases. At each phase, Robo2VLM uses scene and interaction understanding to identify 3D properties of the robot, task goal, and the target object. The properties are used to generate representative VQA queries - images with textural multiple-choice questions - based on spatial, goal-conditioned, and interaction reasoning question templates. We curate Robo2VLM-1, a large-scale in-the-wild dataset with 684,710 questions covering 463 distinct scenes and 3,396 robotic manipulation tasks from 176k real robot trajectories. Results suggest that Robo2VLM-1 can benchmark and improve VLM capabilities in spatial and interaction reasoning.