PyTorchKR Papers

“Visual Question Answering”

총 2,804편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2303.15006

Curriculum Learning for Compositional Visual Reasoning

Visual Question Answering (VQA) is a complex task requiring large datasets and expensive training. Neural Module Networks (NMN) first translate the question to a reasoning path, then follow that path to analyze the image and provide an answer. We propose an NMN method that relies on predefined cross-modal embeddings to ``warm start'' learning on the GQA dataset, then focus on Curriculum Learning (CL) as a way to improve training and make a better use of the data. Several difficulty criteria are employed for defining CL methods. We show that by an appropriate selection of the CL method the cost of training and the amount of training data can be greatly reduced, with a limited impact on the final VQA accuracy. Furthermore, we introduce intermediate losses during training and find that this allows to simplify the CL strategy.

arXiv:2411.02210

One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answering

Vision-Language Models (VLMs) have shown significant promise in Visual Question Answering (VQA) tasks by leveraging web-scale multimodal datasets. However, these models often struggle with continual learning due to catastrophic forgetting when adapting to new tasks. As an effective remedy to mitigate catastrophic forgetting, rehearsal strategy uses the data of past tasks upon learning new task. However, such strategy incurs the need of storing past data, which might not be feasible due to hardware constraints or privacy concerns. In this work, we propose the first data-free method that leverages the language generation capability of a VLM, instead of relying on external models, to produce pseudo-rehearsal data for addressing continual VQA. Our proposal, named as GaB, generates pseudo-rehearsal data by posing previous task questions on new task data. Yet, despite being effective, the distribution of generated questions skews towards the most frequently posed questions due to the limited and task-specific training data. To mitigate this issue, we introduce a pseudo-rehearsal balancing module that aligns the generated data towards the ground-truth data distribution using either the question meta-statistics or an unsupervised clustering method. We evaluate our proposed method on two recent benchmarks, \ie VQACL-VQAv2 and CLOVE-function benchmarks. GaB outperforms all the data-free baselines with substantial improvement in maintaining VQA performance across evolving tasks, while being on-par with methods with access to the past data.

arXiv:2011.03856공식 코드 · Papers with Code

Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles

Many datasets have been shown to contain incidental correlations created by idiosyncrasies in the data collection process. For example, sentence entailment datasets can have spurious word-class correlations if nearly all contradiction sentences contain the word "not", and image recognition datasets can have tell-tale object-background correlations if dogs are always indoors. In this paper, we propose a method that can automatically detect and ignore these kinds of dataset-specific patterns, which we call dataset biases. Our method trains a lower capacity model in an ensemble with a higher capacity model. During training, the lower capacity model learns to capture relatively shallow correlations, which we hypothesize are likely to reflect dataset bias. This frees the higher capacity model to focus on patterns that should generalize better. We ensure the models learn non-overlapping approaches by introducing a novel method to make them conditionally independent. Importantly, our approach does not require the bias to be known in advance. We evaluate performance on synthetic datasets, and four datasets built to penalize models that exploit known biases on textual entailment, visual question answering, and image recognition tasks. We show improvement in all settings, including a 10 point gain on the visual question answering dataset.

arXiv:2502.16486

MQADet: A Plug-and-Play Paradigm for Enhancing Open-Vocabulary Object Detection via Multimodal Question Answering

Open-vocabulary detection (OVD) is a challenging task to detect and classify objects from an unrestricted set of categories, including those unseen during training. Existing open-vocabulary detectors are limited by complex visual-textual misalignment and long-tailed category imbalances, leading to suboptimal performance in challenging scenarios. To address these limitations, we introduce \textbf{MQADet}, a universal paradigm for enhancing existing open-vocabulary detectors by leveraging the cross-modal reasoning capabilities of multimodal large language models (MLLMs). MQADet functions as a plug-and-play solution that integrates seamlessly with pre-trained object detectors without substantial additional training costs. Specifically, we design a novel three-stage Multimodal Question Answering (MQA) pipeline to guide the MLLMs to precisely localize complex textual and visual targets while effectively enhancing the focus of existing object detectors on relevant objects. To validate our approach, we present a new benchmark for evaluating our paradigm on four challenging open-vocabulary datasets, employing three state-of-the-art object detectors as baselines. Experimental results demonstrate that our proposed paradigm significantly improves the performance of existing detectors, particularly in unseen complex categories, across diverse and challenging scenarios. To facilitate future research, we will publicly release our code.

arXiv:1911.11938

Transfer Learning in Visual and Relational Reasoning

Transfer learning has become the de facto standard in computer vision and natural language processing, especially where labeled data is scarce. Accuracy can be significantly improved by using pre-trained models and subsequent fine-tuning. In visual reasoning tasks, such as image question answering, transfer learning is more complex. In addition to transferring the capability to recognize visual features, we also expect to transfer the system's ability to reason. Moreover, for video data, temporal reasoning adds another dimension. In this work, we formalize these unique aspects of transfer learning and propose a theoretical framework for visual reasoning, exemplified by the well-established CLEVR and COG datasets. Furthermore, we introduce a new, end-to-end differentiable recurrent model (SAMNet), which shows state-of-the-art accuracy and better performance in transfer learning on both datasets. The improved performance of SAMNet stems from its capability to decouple the abstract multi-step reasoning from the length of the sequence and its selective attention enabling to store only the question-relevant objects in the external memory.

arXiv:2406.17974공식 코드 · Papers with Code

Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts

Large vision-language models (LVLMs) have recently achieved significant progress, demonstrating strong capabilities in open-world visual understanding. However, it is not yet clear how LVLMs address demographic biases in real life, especially the disparities across attributes such as gender, skin tone, age and race. In this paper, We empirically investigate visual fairness in several mainstream LVLMs by auditing their performance disparities across demographic attributes using public fairness benchmark datasets (e.g., FACET, UTKFace). Our fairness evaluation framework employs direct and single-choice question prompt on visual question-answering/classification tasks. Despite advancements in visual understanding, our zero-shot prompting results show that both open-source and closed-source LVLMs continue to exhibit fairness issues across different prompts and demographic groups. Furthermore, we propose a potential multi-modal Chain-of-thought (CoT) based strategy for bias mitigation, applicable to both open-source and closed-source LVLMs. This approach enhances transparency and offers a scalable solution for addressing fairness, providing a solid foundation for future bias reduction efforts.

arXiv:2104.06645

Jointly Learning Truth-Conditional Denotations and Groundings using Parallel Attention

We present a model that jointly learns the denotations of words together with their groundings using a truth-conditional semantics. Our model builds on the neurosymbolic approach of Mao et al. (2019), learning to ground objects in the CLEVR dataset (Johnson et al., 2017) using a novel parallel attention mechanism. The model achieves state of the art performance on visual question answering, learning to detect and ground objects with question performance as the only training signal. We also show that the model is able to learn flexible non-canonical groundings just by adjusting answers to questions in the training set.

arXiv:1611.07810

A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering

While deep convolutional neural networks frequently approach or exceed human-level performance at benchmark tasks involving static images, extending this success to moving images is not straightforward. Having models which can learn to understand video is of interest for many applications, including content recommendation, prediction, summarization, event/object detection and understanding human visual perception, but many domains lack sufficient data to explore and perfect video models. In order to address the need for a simple, quantitative benchmark for developing and understanding video, we present MovieFIB, a fill-in-the-blank question-answering dataset with over 300,000 examples, based on descriptive video annotations for the visually impaired. In addition to presenting statistics and a description of the dataset, we perform a detailed analysis of 5 different models' predictions, and compare these with human performance. We investigate the relative importance of language, static (2D) visual features, and moving (3D) visual features; the effects of increasing dataset size, the number of frames sampled; and of vocabulary size. We illustrate that: this task is not solvable by a language model alone; our model combining 2D and 3D visual information indeed provides the best result; all models perform significantly worse than human-level. We provide human evaluations for responses given by different models and find that accuracy on the MovieFIB evaluation corresponds well with human judgement. We suggest avenues for improving video models, and hope that the proposed dataset can be useful for measuring and encouraging progress in this very interesting field.

arXiv:2505.20753cs.CV공식 코드 · Papers with Code

Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models

대형 다중모달 모델(Large Multimodal Models, LMMs)은 시각-언어 및 시각 중심 과제에서 뛰어난 성능을 보이나, 복합적 구성 추론(compositional reasoning) 능력 통합에는 한계가 있습니다. 이를 개선하기 위해, 본 연구는 LMM이 내재된 시각 이해 및 그라운딩(grounding) 능력을 활용하여 복잡한 문제를 한 번의 순전파(forwarding)로 인간과 유사한 이해-사고-응답 과정을 수행하도록 하는 통합 시각 추론 메커니즘을 제안합니다. 또한, 33만 4천 개의 시각 지시문 데이터를 구축하여 다양한 장면과 다중 기초 시각 능력을 포함한 학습을 진행하였으며, 이를 통해 개발된 Griffon-R 모델은 복합 시각 추론 벤치마크에서 우수한 성능을 보이고 다중모달 능력 전반을 향상시켰습니다. 데이터와 모델, 코드는 곧 공개될 예정입니다.

arXiv:2406.00980

Selectively Answering Visual Questions

Recently, large multi-modal models (LMMs) have emerged with the capacity to perform vision tasks such as captioning and visual question answering (VQA) with unprecedented accuracy. Applications such as helping the blind or visually impaired have a critical need for precise answers. It is specially important for models to be well calibrated and be able to quantify their uncertainty in order to selectively decide when to answer and when to abstain or ask for clarifications. We perform the first in-depth analysis of calibration methods and metrics for VQA with in-context learning LMMs. Studying VQA on two answerability benchmarks, we show that the likelihood score of visually grounded models is better calibrated than in their text-only counterparts for in-context learning, where sampling based methods are generally superior, but no clear winner arises. We propose Avg BLEU, a calibration score combining the benefits of both sampling and likelihood methods across modalities.

arXiv:2502.11718

"See the World, Discover Knowledge": A Chinese Factuality Evaluation for Large Vision Language Models

The evaluation of factual accuracy in large vision language models (LVLMs) has lagged behind their rapid development, making it challenging to fully reflect these models' knowledge capacity and reliability. In this paper, we introduce the first factuality-based visual question-answering benchmark in Chinese, named ChineseSimpleVQA, aimed at assessing the visual factuality of LVLMs across 8 major topics and 56 subtopics. The key features of this benchmark include a focus on the Chinese language, diverse knowledge types, a multi-hop question construction, high-quality data, static consistency, and easy-to-evaluate through short answers. Moreover, we contribute a rigorous data construction pipeline and decouple the visual factuality into two parts: seeing the world (i.e., object recognition) and discovering knowledge. This decoupling allows us to analyze the capability boundaries and execution mechanisms of LVLMs. Subsequently, we evaluate 34 advanced open-source and closed-source models, revealing critical performance gaps within this field.

arXiv:2307.10405

Generative Visual Question Answering

Multi-modal tasks involving vision and language in deep learning continue to rise in popularity and are leading to the development of newer models that can generalize beyond the extent of their training data. The current models lack temporal generalization which enables models to adapt to changes in future data. This paper discusses a viable approach to creating an advanced Visual Question Answering (VQA) model which can produce successful results on temporal generalization. We propose a new data set, GenVQA, utilizing images and captions from the VQAv2 and MS-COCO dataset to generate new images through stable diffusion. This augmented dataset is then used to test a combination of seven baseline and cutting edge VQA models. Performance evaluation focuses on questions mirroring the original VQAv2 dataset, with the answers having been adjusted to the new images. This paper's purpose is to investigate the robustness of several successful VQA models to assess their performance on future data distributions. Model architectures are analyzed to identify common stylistic choices that improve generalization under temporal distribution shifts. This research highlights the importance of creating a large-scale future shifted dataset. This data can enhance the robustness of VQA models, allowing their future peers to have improved ability to adapt to temporal distribution shifts.

arXiv:2307.15745공식 코드 · Papers with Code

Context-VQA: Towards Context-Aware and Purposeful Visual Question Answering

Visual question answering (VQA) has the potential to make the Internet more accessible in an interactive way, allowing people who cannot see images to ask questions about them. However, multiple studies have shown that people who are blind or have low-vision prefer image explanations that incorporate the context in which an image appears, yet current VQA datasets focus on images in isolation. We argue that VQA models will not fully succeed at meeting people's needs unless they take context into account. To further motivate and analyze the distinction between different contexts, we introduce Context-VQA, a VQA dataset that pairs images with contexts, specifically types of websites (e.g., a shopping website). We find that the types of questions vary systematically across contexts. For example, images presented in a travel context garner 2 times more "Where?" questions, and images on social media and news garner 2.8 and 1.8 times more "Who?" questions than the average. We also find that context effects are especially important when participants can't see the image. These results demonstrate that context affects the types of questions asked and that VQA models should be context-sensitive to better meet people's needs, especially in accessibility settings.

arXiv:2210.14161

Aligning MAGMA by Few-Shot Learning and Finetuning

The goal of vision-language modeling is to allow models to tie language understanding with visual inputs. The aim of this paper is to evaluate and align the Visual Language Model (VLM) called Multimodal Augmentation of Generative Models through Adapter-based finetuning (MAGMA) with human values. MAGMA is a VLM that is capable of image captioning and visual question-answering. We will evaluate its alignment in three different scenarios. To begin, we assess MAGMA's out-of-the-box alignment through the checkpoint provided by Hugging Face. Then, we measure if few-shot learning manages to improve the results. Finally, we finetune the model on aligned examples and evaluate its behavior.

arXiv:2405.19723공식 코드 · Papers with Code

Encoding and Controlling Global Semantics for Long-form Video Question Answering

Seeking answers effectively for long videos is essential to build video question answering (videoQA) systems. Previous methods adaptively select frames and regions from long videos to save computations. However, this fails to reason over the whole sequence of video, leading to sub-optimal performance. To address this problem, we introduce a state space layer (SSL) into multi-modal Transformer to efficiently integrate global semantics of the video, which mitigates the video information loss caused by frame and region selection modules. Our SSL includes a gating unit to enable controllability over the flow of global semantics into visual representations. To further enhance the controllability, we introduce a cross-modal compositional congruence (C^3) objective to encourage global semantics aligned with the question. To rigorously evaluate long-form videoQA capacity, we construct two new benchmarks Ego-QA and MAD-QA featuring videos of considerably long length, i.e. 17.5 minutes and 1.9 hours, respectively. Extensive experiments demonstrate the superiority of our framework on these new as well as existing datasets. The code, model, and data have been made available at https://nguyentthong.github.io/Long_form_VideoQA.

arXiv:2607.02959v1cs.CV

장시간 비디오 질의응답을 위한 비전-언어 모델의 검색 행동 유도

Incentivizing Vision Language Models to Search for Long Video Question Answering

우리는 장시간 비디오 질의응답(LVQA)을 수동적인 단일 패스 인식 작업에서 다중 턴 검색 프로세스로 전환하는 에이전트 프레임워크 VSeek을 소개한다. VSeek은 자연어 기반 검색을 활용하여 장시간 비디오 내 관련 컨텍스트를 식별하고, 강화학습(RL)을 통한 사후 학습으로 LVQA를 위해 목표 검색 쿼리를 공동으로 생성하고 검색된 클립을 추론한다. RL 사후 학습은 수학 및 코드와 같은 기호적 영역의 추론을 혁신했지만, 장시간 비디오 이해에의 적용은 검증된 보상의 부재로 인해 제약을 받아왔다. 검색된 컨텍스트의 관련성을 보장하기 위해, 우리는 개방형 자연어와 이산적 시각 검증을 연결하는 새로운 신경-기호 접근법을 제안한다. 구체적으로, 복잡한 사용자 쿼리를 형식적 시간 논리 명세로 컴파일하여 자연어 질문을 주요 객체 및 활동과 그 시간적 순서 등 필수 원자적 시각 기본 요소들의 명확한 체크리스트로 체계적으로 분해한다. 이렇게 체계적으로 도출된 그라운딩 이벤트는 RL 사후 학습을 위한 핵심 피드백 신호를 제공하며, 결과 정확도에만 의존하지 않고 특정 시각 요소의 성공적인 검색에 기반한 밀도 있고 검증 가능한 보상을 가능하게 한다. 이러한 검증 가능한 증거 탐색 행동을 명시적으로 최적화함으로써, VSeek은 기반 모델 대비 장시간 비디오 이해 벤치마크에서 Pass@1 점수를 최대 8%, Pass@4 점수를 15% 향상시킨다. 코드는 https://utaustin-swarmlab.github.io/VSeek 에서 오픈소스로 공개한다.

arXiv:2210.04563공식 코드 · Papers with Code

Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning

Models for Visual Question Answering (VQA) often rely on the spurious correlations, i.e., the language priors, that appear in the biased samples of training set, which make them brittle against the out-of-distribution (OOD) test data. Recent methods have achieved promising progress in overcoming this problem by reducing the impact of biased samples on model training. However, these models reveal a trade-off that the improvements on OOD data severely sacrifice the performance on the in-distribution (ID) data (which is dominated by the biased samples). Therefore, we propose a novel contrastive learning approach, MMBS, for building robust VQA models by Making the Most of Biased Samples. Specifically, we construct positive samples for contrastive learning by eliminating the information related to spurious correlation from the original training samples and explore several strategies to use the constructed positive samples for training. Instead of undermining the importance of biased samples in model training, our approach precisely exploits the biased samples for unbiased information that contributes to reasoning. The proposed method is compatible with various VQA backbones. We validate our contributions by achieving competitive performance on the OOD dataset VQA-CP v2 while preserving robust performance on the ID dataset VQA v2.

arXiv:2405.09711

STAR: A Benchmark for Situated Reasoning in Real-World Videos

Reasoning in the real world is not divorced from situations. How to capture the present knowledge from surrounding situations and perform reasoning accordingly is crucial and challenging for machine intelligence. This paper introduces a new benchmark that evaluates the situated reasoning ability via situation abstraction and logic-grounded question answering for real-world videos, called Situated Reasoning in Real-World Videos (STAR Benchmark). This benchmark is built upon the real-world videos associated with human actions or interactions, which are naturally dynamic, compositional, and logical. The dataset includes four types of questions, including interaction, sequence, prediction, and feasibility. We represent the situations in real-world videos by hyper-graphs connecting extracted atomic entities and relations (e.g., actions, persons, objects, and relationships). Besides visual perception, situated reasoning also requires structured situation comprehension and logical reasoning. Questions and answers are procedurally generated. The answering logic of each question is represented by a functional program based on a situation hyper-graph. We compare various existing video reasoning models and find that they all struggle on this challenging situated reasoning task. We further propose a diagnostic neuro-symbolic model that can disentangle visual perception, situation abstraction, language understanding, and functional reasoning to understand the challenges of this benchmark.

arXiv:2307.04087공식 코드 · Papers with Code

SVIT: Scaling up Visual Instruction Tuning

Thanks to the emerging of foundation models, the large language and vision models are integrated to acquire the multimodal ability of visual captioning, question answering, etc. Although existing multimodal models present impressive performance of visual understanding and reasoning, their limits are still largely under-explored due to the scarcity of high-quality instruction tuning data. To push the limits of multimodal capability, we Scale up Visual Instruction Tuning (SVIT) by constructing a dataset of 4.2 million visual instruction tuning data including 1.6M conversation question-answer (QA) pairs, 1.6M complex reasoning QA pairs, 1.0M referring QA pairs and 106K detailed image descriptions. Besides the volume, the proposed dataset is also featured by the high quality and rich diversity, which is generated by prompting GPT-4 with the abundant manual annotations of images. We also propose a new data recipe to select subset with better diversity and balance, which evokes model's superior capabilities. Extensive experiments verify that SVIT-v1.5, trained on the proposed dataset, outperforms state-of-the-art Multimodal Large Language Models on popular benchmarks. The data and code are publicly available at https://github.com/BAAI-DCAI/Visual-Instruction-Tuning.

arXiv:2611.22472v1cs.CV

답변이 다를 때 더 살펴보기: 소형 비전-언어 모델이 다른 답을 낸 문항에 시각 토큰 재배정

Disagree to Prune: Reallocating Visual Tokens Where a Small VLM Dissents

시각 토큰 가지치기는 토큰 일부만 남겨 대규모 비전-언어 모델(VLM)의 속도를 높인다. 대부분의 기존 접근법은 토큰 순위를 개선한 뒤 모든 질문에 같은 토큰 예산으로 한 번만 답한다. 순위가 정확하면 가지치기가 영향을 주는 질문은 적고, 남은 토큰만으로 계산한 신호로는 제거한 정보를 거의 파악할 수 없다. 성능이 떨어진 질문 대부분에서는 전체 이미지를 받은 소형 VLM이 가지치기한 모델과 다른 답을 냈다. 소형 모델은 빠진 근거를 볼 수 있기 때문이다. PRUDENT는 먼저 약 절반의 토큰 예산으로 답하고, 아낀 토큰을 이용해 답이 다를 때만 두 배의 예산으로 다시 답한다. 다시 답하는 것이 유리한 조건과 소형 VLM이 전체 이미지를 읽어야 하는 이유를 설명한다. 벤치마크 18개에서 평균 시각 토큰 예산이 같을 때 PRUDENT는 기존 가지치기 방법 여덟 개보다 성능이 높았다. 예산이 줄어들수록 격차가 커졌으며, InternVL의 다섯 가지 모델 크기에서 이점이 유지됐다. 다른 두 모델 계열에서도 평균 성능이 가장 높았다. 소형 VLM과 답이 다른 경우는 여덟 방법 모두에서 발생한 가지치기 실패의 대부분을 찾아냈다. 가장 우수한 기존 방법 두 개도 자체 토큰 순위를 유지한 채 답이 다를 때 다시 답하도록 하자 성능이 높아졌다. 다른 검증기를 써도 개선 폭은 비슷했으며, 첫 답변이 손상된 곳에 추가 토큰을 사용할 수 있었다. 토큰의 순위뿐 아니라 가지치기한 답 중 어떤 답을 신뢰할지 판단하는 것도 시각 토큰 가지치기를 개선할 수 있음을 시사한다. 코드와 데이터는 https://github.com/LiZaiyuan0619/PRUDENT 에 공개되어 있다.