PyTorchKR Papers

“Visual Question Answering”

총 2,804편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2412.14880

Multimodal Hypothetical Summary for Retrieval-based Multi-image Question Answering

Retrieval-based multi-image question answering (QA) task involves retrieving multiple question-related images and synthesizing these images to generate an answer. Conventional "retrieve-then-answer" pipelines often suffer from cascading errors because the training objective of QA fails to optimize the retrieval stage. To address this issue, we propose a novel method to effectively introduce and reference retrieved information into the QA. Given the image set to be retrieved, we employ a multimodal large language model (visual perspective) and a large language model (textual perspective) to obtain multimodal hypothetical summary in question-form and description-form. By combining visual and textual perspectives, MHyS captures image content more specifically and replaces real images in retrieval, which eliminates the modality gap by transforming into text-to-text retrieval and helps improve retrieval. To more advantageously introduce retrieval with QA, we employ contrastive learning to align queries (questions) with MHyS. Moreover, we propose a coarse-to-fine strategy for calculating both sentence-level and word-level similarity scores, to further enhance retrieval and filter out irrelevant details. Our approach achieves a 3.7% absolute improvement over state-of-the-art methods on RETVQA and a 14.5% improvement over CLIP. Comprehensive experiments and detailed ablation studies demonstrate the superiority of our method.

arXiv:2410.22029공식 코드 · Papers with Code

Are VLMs Really Blind

Vision Language Models excel in handling a wide range of complex tasks, including Optical Character Recognition (OCR), Visual Question Answering (VQA), and advanced geometric reasoning. However, these models fail to perform well on low-level basic visual tasks which are especially easy for humans. Our goal in this work was to determine if these models are truly "blind" to geometric reasoning or if there are ways to enhance their capabilities in this area. Our work presents a novel automatic pipeline designed to extract key information from images in response to specific questions. Instead of just relying on direct VQA, we use question-derived keywords to create a caption that highlights important details in the image related to the question. This caption is then used by a language model to provide a precise answer to the question without requiring external fine-tuning.

arXiv:2405.10736

StackOverflowVQA: Stack Overflow Visual Question Answering Dataset

In recent years, people have increasingly used AI to help them with their problems by asking questions on different topics. One of these topics can be software-related and programming questions. In this work, we focus on the questions which need the understanding of images in addition to the question itself. We introduce the StackOverflowVQA dataset, which includes questions from StackOverflow that have one or more accompanying images. This is the first VQA dataset that focuses on software-related questions and contains multiple human-generated full-sentence answers. Additionally, we provide a baseline for answering the questions with respect to images in the introduced dataset using the GIT model. All versions of the dataset are available at https://huggingface.co/mirzaei2114.

arXiv:2407.01907

The Solution for the ICCV 2023 Perception Test Challenge 2023 -- Task 6 -- Grounded videoQA

In this paper, we introduce a grounded video question-answering solution. Our research reveals that the fixed official baseline method for video question answering involves two main steps: visual grounding and object tracking. However, a significant challenge emerges during the initial step, where selected frames may lack clearly identifiable target objects. Furthermore, single images cannot address questions like "Track the container from which the person pours the first time." To tackle this issue, we propose an alternative two-stage approach:(1) First, we leverage the VALOR model to answer questions based on video information.(2) concatenate the answered questions with their respective answers. Finally, we employ TubeDETR to generate bounding boxes for the targets.

arXiv:2410.12705공식 코드 · Papers with Code

WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines

Vision Language Models (VLMs) often struggle with culture-specific knowledge, particularly in languages other than English and in underrepresented cultural contexts. To evaluate their understanding of such knowledge, we introduce WorldCuisines, a massive-scale benchmark for multilingual and multicultural, visually grounded language understanding. This benchmark includes a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects, spanning 9 language families and featuring over 1 million data points, making it the largest multicultural VQA benchmark to date. It includes tasks for identifying dish names and their origins. We provide evaluation datasets in two sizes (12k and 60k instances) alongside a training dataset (1 million instances). Our findings show that while VLMs perform better with correct location context, they struggle with adversarial contexts and predicting specific regional cuisines and languages. To support future research, we release a knowledge base with annotated food entries and images along with the VQA data.

arXiv:2603.09541v2cs.CV

동적인 사람 밀집 장면에서의 체화 질의응답을 위한 메모리 효율적 능동 지각

Memory-Efficient Active Perception for Embodied Question Answering in Dynamic Human-Populated Scenes

체화 질의응답(EQA)은 능동적 시각 지각, 시각-언어 추론, 순차적 의사결정을 결합하며, 시각적 증거를 안정적으로 축적할 수 있는 시간적으로 안정된 환경에서 전통적으로 평가되어 왔다. 그러나 동적인 사람 밀집 장면과 같은 현실적인 멀티미디어 환경에서는 움직이는 사람과 일시적 가림이 사람에 의한 지각 변동을 일으켜, 과제와 관련된 멀티모달 단서가 일시적이고 시점에 의존적이며 개별 뷰에서 불완전하게만 포착된다. 기존의 저장 후 검색 전략은 중복된 시각적 증거를 생성하고 높은 추론 비용을 초래하므로 이러한 변동에 적합하지 않다. 이는 체화 에이전트에게 두 가지 핵심 문제, 즉 시점에 의존적인 모호성의 해소와 효율적 추론을 위한 간결한 증거의 유지를 드러낸다. 이러한 과제를 해결하기 위해, 우리는 EQA를 위한 학습이 필요 없는 능동 증거 획득 프레임워크인 DIVRR(Dynamic-Informed View Refinement with Relevance-guided Rotation and adaptive memory selection)을 제시한다. DIVRR은 질문 조건부 방 안내와 관련성 기반 뷰 증강을 결합하여 탐색과 추가 관찰을 유도한다. 선택적 메모리 수용은 뷰 증강과 동일한 관련성 신호를 사용해 높은 점수의 증거를 유지함으로써, 검색된 시각적 맥락을 간결하게 유지하면서 저장되는 항목 수를 줄인다. 사람에 의한 지각 변동 하에서 EQA를 평가하기 위해, 우리는 Habitat에서 구축된, 동일한 장면 배치를 공유하는 동적 및 정적 부분집합으로 구성된 동적 사람 밀집 EQA 데이터셋인 DynHuman-EQA를 소개한다. DynHuman-EQA와 HM-EQA에서의 실험 결과, DIVRR은 정확도와 메모리 효율 모두에서 기존 기준 모델을 능가한다. 이득은 동적인 사람 밀집 장면에서 가장 두드러진다.

arXiv:2505.18915공식 코드 · Papers with Code

Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering

Vision-Language Models (VLMs) have shown promise in various 2D visual tasks, yet their readiness for 3D clinical diagnosis remains unclear due to stringent demands for recognition precision, reasoning ability, and domain knowledge. To systematically evaluate these dimensions, we present DeepTumorVQA, a diagnostic visual question answering (VQA) benchmark targeting abdominal tumors in CT scans. It comprises 9,262 CT volumes (3.7M slices) from 17 public datasets, with 395K expert-level questions spanning four categories: Recognition, Measurement, Visual Reasoning, and Medical Reasoning. DeepTumorVQA introduces unique challenges, including small tumor detection and clinical reasoning across 3D anatomy. Benchmarking four advanced VLMs (RadFM, M3D, Merlin, CT-CHAT), we find current models perform adequately on measurement tasks but struggle with lesion recognition and reasoning, and are still not meeting clinical needs. Two key insights emerge: (1) large-scale multimodal pretraining plays a crucial role in DeepTumorVQA testing performance, making RadFM stand out among all VLMs. (2) Our dataset exposes critical differences in VLM components, where proper image preprocessing and design of vision modules significantly affect 3D perception. To facilitate medical multimodal research, we have released DeepTumorVQA as a rigorous benchmark: https://github.com/Schuture/DeepTumorVQA.

arXiv:2608.01185v1cs.CV

3DZip: 3차원 질의응답을 위한 공간 인식 특징 다양성 기반 토큰 압축

3DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answering

최근의 3차원 비전-언어 모델(3D VLM)은 2D 시각적 특징(feature)을 월드 좌표계로 투영하여 기하학 인식 토큰을 구성함으로써, 3D 질의응답과 같은 태스크를 위한 공간적 추론을 가능하게 합니다. 그러나 이러한 설계는 장면당 수천 개의 토큰(token)을 생성하여 상당한 연산 및 메모리 오버헤드를 초래합니다. 2D VLM에서는 토큰 압축이 광범위하게 연구되어 왔지만, 기존 접근 방식은 3D 토큰의 구조화된 공간적 특성을 간과하는 의미론적 관련성이나 어텐션(attention) 기반 선택에 의존합니다. 또한, 공간적 집계 후에도 객체 수준의 토큰 불균형이 지속되기 때문에 3D 표현의 중복성은 공간적 근접성만으로는 해결될 수 없습니다. 이를 해결하기 위해, 먼저 포인트 수준의 중복성을 제거하기 위해 대략적인 복셀화(voxelization)를 적용한 다음, 행렬식 점 과정(Determinantal Point Process)을 통해 특징 공간 다양성을 기반으로 앵커 토큰을 선택하고, 마지막으로 기하학적 일관성을 보존하기 위해 공간적 제약 하에서 나머지 토큰을 병합하는 3단계 토큰 압축 프레임워크인 3DZip을 제안합니다. 세 가지 3D 질의응답 벤치마크에 대한 실험을 통해 3DZip이 기존 압축 방법들을 일관되게 능가하며, 단 128개의 토큰만으로 원래 성능의 94.7%를 유지하고 1.92배 빠른 추론(inference) 속도를 달성함을 입증합니다.

arXiv:2412.00102

ElectroVizQA: How well do Multi-modal LLMs perform in Electronics Visual Question Answering?

Multi-modal Large Language Models (MLLMs) are gaining significant attention for their ability to process multi-modal data, providing enhanced contextual understanding of complex problems. MLLMs have demonstrated exceptional capabilities in tasks such as Visual Question Answering (VQA); however, they often struggle with fundamental engineering problems, and there is a scarcity of specialized datasets for training on topics like digital electronics. To address this gap, we propose a benchmark dataset called ElectroVizQA specifically designed to evaluate MLLMs' performance on digital electronic circuit problems commonly found in undergraduate curricula. This dataset, the first of its kind tailored for the VQA task in digital electronics, comprises approximately 626 visual questions, offering a comprehensive overview of digital electronics topics. This paper rigorously assesses the extent to which MLLMs can understand and solve digital electronic circuit questions, providing insights into their capabilities and limitations within this specialized domain. By introducing this benchmark dataset, we aim to motivate further research and development in the application of MLLMs to engineering education, ultimately bridging the performance gap and enhancing the efficacy of these models in technical fields.

arXiv:2501.04568

Feedback-Driven Vision-Language Alignment with Minimal Human Supervision

Vision-language models (VLMs) have demonstrated remarkable potential in integrating visual and linguistic information, but their performance is often constrained by the need for extensive, high-quality image-text training data. Curation of these image-text pairs is both time-consuming and computationally expensive. To address this challenge, we introduce SVP (Sampling-based Visual Projection), a novel framework that enhances vision-language alignment without relying on manually curated text-image pairs or preference annotation. SVP leverages a small set of manually selected images, self-captioning and a pre-trained grounding model as a feedback mechanism to elicit latent information in VLMs. We evaluate our approach across six key areas: captioning, referring, visual question answering, multitasking, hallucination control, and object recall. Results demonstrate significant improvements, including a 14 % average improvement in captioning tasks, up to 12 % increase in object recall, and significantly reduced hallucinations, while maintaining question-answering capabilities. Using SVP, a small VLM achieves hallucination reductions similar to a model five times larger, while a VLM with initially poor referring capabilities more than doubles its performance, approaching parity with a model twice its size.

arXiv:1907.12861

LEAF-QA: Locate, Encode & Attend for Figure Question Answering

We introduce LEAF-QA, a comprehensive dataset of $250,000$ densely annotated figures/charts, constructed from real-world open data sources, along with ~2 million question-answer (QA) pairs querying the structure and semantics of these charts. LEAF-QA highlights the problem of multimodal QA, which is notably different from conventional visual QA (VQA), and has recently gained interest in the community. Furthermore, LEAF-QA is significantly more complex than previous attempts at chart QA, viz. FigureQA and DVQA, which present only limited variations in chart data. LEAF-QA being constructed from real-world sources, requires a novel architecture to enable question answering. To this end, LEAF-Net, a deep architecture involving chart element localization, question and answer encoding in terms of chart elements, and an attention network is proposed. Different experiments are conducted to demonstrate the challenges of QA on LEAF-QA. The proposed architecture, LEAF-Net also considerably advances the current state-of-the-art on FigureQA and DVQA.

arXiv:2412.09230

Foundation Models and Adaptive Feature Selection: A Synergistic Approach to Video Question Answering

This paper tackles the intricate challenge of video question-answering (VideoQA). Despite notable progress, current methods fall short of effectively integrating questions with video frames and semantic object-level abstractions to create question-aware video representations. We introduce Local-Global Question Aware Video Embedding (LGQAVE), which incorporates three major innovations to integrate multi-modal knowledge better and emphasize semantic visual concepts relevant to specific questions. LGQAVE moves beyond traditional ad-hoc frame sampling by utilizing a cross-attention mechanism that precisely identifies the most relevant frames concerning the questions. It captures the dynamics of objects within these frames using distinct graphs, grounding them in question semantics with the miniGPT model. These graphs are processed by a question-aware dynamic graph transformer (Q-DGT), which refines the outputs to develop nuanced global and local video representations. An additional cross-attention module integrates these local and global embeddings to generate the final video embeddings, which a language model uses to generate answers. Extensive evaluations across multiple benchmarks demonstrate that LGQAVE significantly outperforms existing models in delivering accurate multi-choice and open-ended answers.

arXiv:2205.05019공식 코드 · Papers with Code

Learning to Answer Visual Questions from Web Videos

Recent methods for visual question answering rely on large-scale annotated datasets. Manual annotation of questions and answers for videos, however, is tedious, expensive and prevents scalability. In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision. We leverage a question generation transformer trained on text data and use it to generate question-answer pairs from transcribed video narrations. Given narrated videos, we then automatically generate the HowToVQA69M dataset with 69M video-question-answer triplets. To handle the open vocabulary of diverse answers in this dataset, we propose a training procedure based on a contrastive loss between a video-question multi-modal transformer and an answer transformer. We introduce the zero-shot VideoQA task and the VideoQA feature probe evaluation setting and show excellent results, in particular for rare answers. Furthermore, our method achieves competitive results on MSRVTT-QA, ActivityNet-QA, MSVD-QA and How2QA datasets. We also show that our VideoQA dataset generation approach generalizes to another source of web video and text data. We use our method to generate the WebVidVQA3M dataset from the WebVid dataset, i.e., videos with alt-text annotations, and show its benefits for training VideoQA models. Finally, for a detailed evaluation we introduce iVQA, a new VideoQA dataset with reduced language bias and high-quality manual annotations. Code, datasets and trained models are available at https://antoyang.github.io/just-ask.html

arXiv:1811.11903

Visual Question Answering as Reading Comprehension

Visual question answering (VQA) demands simultaneous comprehension of both the image visual content and natural language questions. In some cases, the reasoning needs the help of common sense or general knowledge which usually appear in the form of text. Current methods jointly embed both the visual information and the textual feature into the same space. However, how to model the complex interactions between the two different modalities is not an easy task. In contrast to struggling on multimodal feature fusion, in this paper, we propose to unify all the input information by natural language so as to convert VQA into a machine reading comprehension problem. With this transformation, our method not only can tackle VQA datasets that focus on observation based questions, but can also be naturally extended to handle knowledge-based VQA which requires to explore large-scale external knowledge base. It is a step towards being able to exploit large volumes of text and natural language processing techniques to address VQA problem. Two types of models are proposed to deal with open-ended VQA and multiple-choice VQA respectively. We evaluate our models on three VQA benchmarks. The comparable performance with the state-of-the-art demonstrates the effectiveness of the proposed method.

arXiv:1902.00038공식 코드 · Papers with Code

BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection

Multimodal representation learning is gaining more and more interest within the deep learning community. While bilinear models provide an interesting framework to find subtle combination of modalities, their number of parameters grows quadratically with the input dimensions, making their practical implementation within classical deep learning pipelines challenging. In this paper, we introduce BLOCK, a new multimodal fusion based on the block-superdiagonal tensor decomposition. It leverages the notion of block-term ranks, which generalizes both concepts of rank and mode ranks for tensors, already used for multimodal fusion. It allows to define new ways for optimizing the tradeoff between the expressiveness and complexity of the fusion model, and is able to represent very fine interactions between modalities while maintaining powerful mono-modal representations. We demonstrate the practical interest of our fusion model by using BLOCK for two challenging tasks: Visual Question Answering (VQA) and Visual Relationship Detection (VRD), where we design end-to-end learnable architectures for representing relevant interactions between modalities. Through extensive experiments, we show that BLOCK compares favorably with respect to state-of-the-art multimodal fusion models for both VQA and VRD tasks. Our code is available at https://github.com/Cadene/block.bootstrap.pytorch.

arXiv:2210.03941공식 코드 · Papers with Code

Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modeling

While recent large-scale video-language pre-training made great progress in video question answering, the design of spatial modeling of video-language models is less fine-grained than that of image-language models; existing practices of temporal modeling also suffer from weak and noisy alignment between modalities. To learn fine-grained visual understanding, we decouple spatial-temporal modeling and propose a hybrid pipeline, Decoupled Spatial-Temporal Encoders, integrating an image- and a video-language encoder. The former encodes spatial semantics from larger but sparsely sampled frames independently of time, while the latter models temporal dynamics at lower spatial but higher temporal resolution. To help the video-language model learn temporal relations for video QA, we propose a novel pre-training objective, Temporal Referring Modeling, which requires the model to identify temporal positions of events in video sequences. Extensive experiments demonstrate that our model outperforms previous work pre-trained on orders of magnitude larger datasets.

arXiv:2205.14444

Visual Superordinate Abstraction for Robust Concept Learning

Concept learning constructs visual representations that are connected to linguistic semantics, which is fundamental to vision-language tasks. Although promising progress has been made, existing concept learners are still vulnerable to attribute perturbations and out-of-distribution compositions during inference. We ascribe the bottleneck to a failure of exploring the intrinsic semantic hierarchy of visual concepts, e.g. \{red, blue,...\} $\in$ `color' subspace yet cube $\in$ `shape'. In this paper, we propose a visual superordinate abstraction framework for explicitly modeling semantic-aware visual subspaces (i.e. visual superordinates). With only natural visual question answering data, our model first acquires the semantic hierarchy from a linguistic view, and then explores mutually exclusive visual superordinates under the guidance of linguistic hierarchy. In addition, a quasi-center visual concept clustering and a superordinate shortcut learning schemes are proposed to enhance the discrimination and independence of concepts within each visual superordinate. Experiments demonstrate the superiority of the proposed framework under diverse settings, which increases the overall answering accuracy relatively by 7.5\% on reasoning with perturbations and 15.6\% on compositional generalization tests.

arXiv:2407.15589

Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models

Object-centric (OC) representations, which represent the state of a visual scene by modeling it as a composition of objects, have the potential to be used in various downstream tasks to achieve systematic compositional generalization and facilitate reasoning. However, these claims have not been thoroughly analyzed yet. Recently, foundation models have demonstrated unparalleled capabilities across diverse domains from language to computer vision, marking them as a potential cornerstone of future research for a multitude of computational tasks. In this paper, we conduct an extensive empirical study on representation learning for downstream Visual Question Answering (VQA), which requires an accurate compositional understanding of the scene. We thoroughly investigate the benefits and trade-offs of OC models and alternative approaches including large pre-trained foundation models on both synthetic and real-world data, and demonstrate a viable way to achieve the best of both worlds. The extensiveness of our study, encompassing over 600 downstream VQA models and 15 different types of upstream representations, also provides several additional insights that we believe will be of interest to the community at large.

arXiv:2506.22880공식 코드 · Papers with Code

Decoupled Seg Tokens Make Stronger Reasoning Video Segmenter and Grounder

Existing video segmenter and grounder approaches, exemplified by Sa2VA, directly fuse features within segmentation models. This often results in an undesirable entanglement of dynamic visual information and static semantics, thereby degrading segmentation accuracy. To systematically mitigate this issue, we propose DeSa2VA, a decoupling-enhanced prompting scheme integrating text pre-training and a linear decoupling module to address the information processing limitations inherent in SAM-2. Specifically, first, we devise a pre-training paradigm that converts textual ground-truth labels into point-level prompts while generating corresponding text masks. These masks are refined through a hybrid loss function to strengthen the model's semantic grounding capabilities. Next, we employ linear projection to disentangle hidden states that generated by a large language model into distinct textual and visual feature subspaces. Finally, a dynamic mask fusion strategy synergistically combines these decoupled features through triple supervision from predicted text/visual masks and ground-truth annotations. Extensive experiments demonstrate state-of-the-art performance across diverse tasks, including image segmentation, image question answering, video segmentation, and video question answering. Our codes are available at https://github.com/longmalongma/DeSa2VA.

arXiv:2608.10864v3cs.CV

비전-언어 모델의 공간 추론을 위한 다중 뷰 관계 증류

Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Models

비전-언어 모델(VLM)은 강력한 이미지 및 비디오 이해 능력을 달성했으나, 시각-공간 표현은 기하학적으로 여전히 취약하여 임바디드 AI, 로보틱스, 자율주행에 필요한 공간 추론에서 실패를 초래한다. 기존의 기하 기반 접지(geometry grounding) 접근법은 공간 질의응답으로 VLM을 미세 조정하거나(이는 허위 시각 표현을 고착시킬 수 있다), 대규모 기하 기반 비전 모델의 특징을 융합하는데(이는 추론 시 모델 크기를 크게 증가시킨다), 둘 중 하나이다. 기하 기반 비전 모델로부터의 지식 증류는 대안이 되지만, 다중 뷰 교사 특징을 직접 일치시키면 시각 표현과 텍스트 표현 사이의 사전학습된 정렬이 깨져 객체 및 언어 의미 능력이 저하될 수 있다. 본 논문은 교사 특징 자체 대신 뷰 간 패치 단위 코사인 유사도를 증류하는 다중 뷰 관계 증류(MVRD)를 제안한다. 이러한 관계는 공간 이해에 충분한 기하학적 대응을 인코딩하면서도 학생 표현을 미결정 상태로 남겨, 사전학습된 비전-언어 공간에 가깝게 유지될 수 있게 한다. 대표적인 여러 VLM에서 MVRD는 시각-공간 추론을 향상시키며, 지도 미세 조정과 특징 증류를 능가하고, 추가 파라미터 수와 지연 시간은 크게 적으면서 특징 융합 방법에 근접한다. MVRD가 언어 정렬을 유지하면서 시각 표현을 더 기하학적으로 만들고, 객체 접지, 밀집 캡셔닝, 질의응답과 같은 3D 장면 이해 과제로 일반화됨을 보인다. 코드는 https://github.com/kietngt00/MVRD 에서 확인할 수 있다.