PyTorchKR Papers

“Visual Question Answering”

총 2,807편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2404.04627

Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement

Visual program synthesis is a promising approach to exploit the reasoning abilities of large language models for compositional computer vision tasks. Previous work has used few-shot prompting with frozen LLMs to synthesize visual programs. Training an LLM to write better visual programs is an attractive prospect, but it is unclear how to accomplish this. No dataset of visual programs for training exists, and acquisition of a visual program dataset cannot be easily crowdsourced due to the need for expert annotators. To get around the lack of direct supervision, we explore improving the program synthesis abilities of an LLM using feedback from interactive experience. We propose a method where we exploit existing annotations for a vision-language task to improvise a coarse reward signal for that task, treat the LLM as a policy, and apply reinforced self-training to improve the visual program synthesis ability of the LLM for that task. We describe a series of experiments on object detection, compositional visual question answering, and image-text retrieval, and show that in each case, the self-trained LLM outperforms or performs on par with few-shot frozen LLMs that are an order of magnitude larger. Website: https://zaidkhan.me/ViReP

arXiv:2107.06383공식 코드 · Papers with Code

How Much Can CLIP Benefit Vision-and-Language Tasks?

Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks. We release our code at https://github.com/clip-vil/CLIP-ViL.

arXiv:2305.02317

Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings

Recent advances in large language models elicit reasoning in a chain-of-thought that allows models to decompose problems in a human-like fashion. Though this paradigm improves multi-step reasoning ability in language models, it is limited by being unimodal and applied mainly to question-answering tasks. We claim that incorporating visual augmentation into reasoning is essential, especially for complex, imaginative tasks. Consequently, we introduce VCoT, a novel method that leverages chain-of-thought prompting with vision-language grounding to recursively bridge the logical gaps within sequential data. Our method uses visual guidance to generate synthetic multimodal infillings that add consistent and novel information to reduce the logical gaps for downstream tasks that can benefit from temporal reasoning, as well as provide interpretability into models' multi-step reasoning. We apply VCoT to the Visual Storytelling and WikiHow summarization datasets and demonstrate through human evaluation that VCoT offers novel and consistent synthetic data augmentation beating chain-of-thought baselines, which can be used to enhance downstream performance.

arXiv:2409.05395공식 코드 · Papers with Code

Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling

This study explores replacing Transformers in Visual Language Models (VLMs) with Mamba, a recent structured state space model (SSM) that demonstrates promising performance in sequence modeling. We test models up to 3B parameters under controlled conditions, showing that Mamba-based VLMs outperforms Transformers-based VLMs in captioning, question answering, and reading comprehension. However, we find that Transformers achieve greater performance in visual grounding and the performance gap widens with scale. We explore two hypotheses to explain this phenomenon: 1) the effect of task-agnostic visual encoding on the updates of the hidden states, and 2) the difficulty in performing visual grounding from the perspective of in-context multimodal retrieval. Our results indicate that a task-aware encoding yields minimal performance gains on grounding, however, Transformers significantly outperform Mamba at in-context multimodal retrieval. Overall, Mamba shows promising performance on tasks where the correct output relies on a summary of the image but struggles when retrieval of explicit information from the context is required.

arXiv:2001.04732공식 코드 · Papers with Code

Fine-grained Image Classification and Retrieval by Combining Visual and Locally Pooled Textual Features

Text contained in an image carries high-level semantics that can be exploited to achieve richer image understanding. In particular, the mere presence of text provides strong guiding content that should be employed to tackle a diversity of computer vision tasks such as image retrieval, fine-grained classification, and visual question answering. In this paper, we address the problem of fine-grained classification and image retrieval by leveraging textual information along with visual cues to comprehend the existing intrinsic relation between the two modalities. The novelty of the proposed model consists of the usage of a PHOC descriptor to construct a bag of textual words along with a Fisher Vector Encoding that captures the morphology of text. This approach provides a stronger multimodal representation for this task and as our experiments demonstrate, it achieves state-of-the-art results on two different tasks, fine-grained classification and image retrieval.

arXiv:2401.13313공식 코드 · Papers with Code

InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions

We study the problem of completing various visual document understanding (VDU) tasks, e.g., question answering and information extraction, on real-world documents through human-written instructions. To this end, we propose InstructDoc, the first large-scale collection of 30 publicly available VDU datasets, each with diverse instructions in a unified format, which covers a wide range of 12 tasks and includes open document types/formats. Furthermore, to enhance the generalization performance on VDU tasks, we design a new instruction-based document reading and understanding model, InstructDr, that connects document images, image encoders, and large language models (LLMs) through a trainable bridging module. Experiments demonstrate that InstructDr can effectively adapt to new VDU datasets, tasks, and domains via given instructions and outperforms existing multimodal LLMs and ChatGPT without specific training.

arXiv:2408.08632공식 코드 · Papers with Code

A Survey on Benchmarks of Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) are gaining increasing popularity in both academia and industry due to their remarkable performance in various applications such as visual question answering, visual perception, understanding, and reasoning. Over the past few years, significant efforts have been made to examine MLLMs from multiple perspectives. This paper presents a comprehensive review of 200 benchmarks and evaluations for MLLMs, focusing on (1)perception and understanding, (2)cognition and reasoning, (3)specific domains, (4)key capabilities, and (5)other modalities. Finally, we discuss the limitations of the current evaluation methods for MLLMs and explore promising future directions. Our key argument is that evaluation should be regarded as a crucial discipline to support the development of MLLMs better. For more details, please visit our GitHub repository: https://github.com/swordlidev/Evaluation-Multimodal-LLMs-Survey.

arXiv:2502.18725

Talking to the brain: Using Large Language Models as Proxies to Model Brain Semantic Representation

Traditional psychological experiments utilizing naturalistic stimuli face challenges in manual annotation and ecological validity. To address this, we introduce a novel paradigm leveraging multimodal large language models (LLMs) as proxies to extract rich semantic information from naturalistic images through a Visual Question Answering (VQA) strategy for analyzing human visual semantic representation. LLM-derived representations successfully predict established neural activity patterns measured by fMRI (e.g., faces, buildings), validating its feasibility and revealing hierarchical semantic organization across cortical regions. A brain semantic network constructed from LLM-derived representations identifies meaningful clusters reflecting functional and contextual associations. This innovative methodology offers a powerful solution for investigating brain semantic organization with naturalistic stimuli, overcoming limitations of traditional annotation methods and paving the way for more ecologically valid explorations of human cognition.

arXiv:2103.15974

Domain-robust VQA with diverse datasets and methods but no target labels

The observation that computer vision methods overfit to dataset specifics has inspired diverse attempts to make object recognition models robust to domain shifts. However, similar work on domain-robust visual question answering methods is very limited. Domain adaptation for VQA differs from adaptation for object recognition due to additional complexity: VQA models handle multimodal inputs, methods contain multiple steps with diverse modules resulting in complex optimization, and answer spaces in different datasets are vastly different. To tackle these challenges, we first quantify domain shifts between popular VQA datasets, in both visual and textual space. To disentangle shifts between datasets arising from different modalities, we also construct synthetic shifts in the image and question domains separately. Second, we test the robustness of different families of VQA methods (classic two-stream, transformer, and neuro-symbolic methods) to these shifts. Third, we test the applicability of existing domain adaptation methods and devise a new one to bridge VQA domain gaps, adjusted to specific VQA models. To emulate the setting of real-world generalization, we focus on unsupervised domain adaptation and the open-ended classification task formulation.

arXiv:2306.05523

FACTIFY3M: A Benchmark for Multimodal Fact Verification with Explainability through 5W Question-Answering

Combating disinformation is one of the burning societal crises -- about 67% of the American population believes that disinformation produces a lot of uncertainty, and 10% of them knowingly propagate disinformation. Evidence shows that disinformation can manipulate democratic processes and public opinion, causing disruption in the share market, panic and anxiety in society, and even death during crises. Therefore, disinformation should be identified promptly and, if possible, mitigated. With approximately 3.2 billion images and 720,000 hours of video shared online daily on social media platforms, scalable detection of multimodal disinformation requires efficient fact verification. Despite progress in automatic text-based fact verification (e.g., FEVER, LIAR), the research community lacks substantial effort in multimodal fact verification. To address this gap, we introduce FACTIFY 3M, a dataset of 3 million samples that pushes the boundaries of the domain of fact verification via a multimodal fake news dataset, in addition to offering explainability through the concept of 5W question-answering. Salient features of the dataset include: (i) textual claims, (ii) ChatGPT-generated paraphrased claims, (iii) associated images, (iv) stable diffusion-generated additional images (i.e., visual paraphrases), (v) pixel-level image heatmap to foster image-text explainability of the claim, (vi) 5W QA pairs, and (vii) adversarial fake news stories.

arXiv:2212.08283

SceneGATE: Scene-Graph based co-Attention networks for TExt visual question answering

Most TextVQA approaches focus on the integration of objects, scene texts and question words by a simple transformer encoder. But this fails to capture the semantic relations between different modalities. The paper proposes a Scene Graph based co-Attention Network (SceneGATE) for TextVQA, which reveals the semantic relations among the objects, Optical Character Recognition (OCR) tokens and the question words. It is achieved by a TextVQA-based scene graph that discovers the underlying semantics of an image. We created a guided-attention module to capture the intra-modal interplay between the language and the vision as a guidance for inter-modal interactions. To make explicit teaching of the relations between the two modalities, we proposed and integrated two attention modules, namely a scene graph-based semantic relation-aware attention and a positional relation-aware attention. We conducted extensive experiments on two benchmark datasets, Text-VQA and ST-VQA. It is shown that our SceneGATE method outperformed existing ones because of the scene graph and its attention modules.

arXiv:2303.05952

Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning

Contrastive loss has been increasingly used in learning representations from multiple modalities. In the limit, the nature of the contrastive loss encourages modalities to exactly match each other in the latent space. Yet it remains an open question how the modality alignment affects the downstream task performance. In this paper, based on an information-theoretic argument, we first prove that exact modality alignment is sub-optimal in general for downstream prediction tasks. Hence we advocate that the key of better performance lies in meaningful latent modality structures instead of perfect modality alignment. To this end, we propose three general approaches to construct latent modality structures. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization. Extensive experiments are conducted on two popular multi-modal representation learning frameworks: the CLIP-based two-tower model and the ALBEF-based fusion model. We test our model on a variety of tasks including zero/few-shot image classification, image-text retrieval, visual question answering, visual reasoning, and visual entailment. Our method achieves consistent improvements over existing methods, demonstrating the effectiveness and generalizability of our proposed approach on latent modality structure regularization.

arXiv:2007.08814공식 코드 · Papers with Code

Visual Relation Grounding in Videos

In this paper, we explore a novel task named visual Relation Grounding in Videos (vRGV). The task aims at spatio-temporally localizing the given relations in the form of subject-predicate-object in the videos, so as to provide supportive visual facts for other high-level video-language tasks (e.g., video-language grounding and video question answering). The challenges in this task include but not limited to: (1) both the subject and object are required to be spatio-temporally localized to ground a query relation; (2) the temporal dynamic nature of visual relations in videos is difficult to capture; and (3) the grounding should be achieved without any direct supervision in space and time. To ground the relations, we tackle the challenges by collaboratively optimizing two sequences of regions over a constructed hierarchical spatio-temporal region graph through relation attending and reconstruction, in which we further propose a message passing mechanism by spatial attention shifting between visual entities. Experimental results demonstrate that our model can not only outperform baseline approaches significantly, but also produces visually meaningful facts to support visual grounding. (Code is available at https://github.com/doc-doc/vRGV).

arXiv:2503.07635공식 코드 · Papers with Code

Cross-modal Causal Relation Alignment for Video Question Grounding

Video question grounding (VideoQG) requires models to answer the questions and simultaneously infer the relevant video segments to support the answers. However, existing VideoQG methods usually suffer from spurious cross-modal correlations, leading to a failure to identify the dominant visual scenes that align with the intended question. Moreover, vision-language models exhibit unfaithful generalization performance and lack robustness on challenging downstream tasks such as VideoQG. In this work, we propose a novel VideoQG framework named Cross-modal Causal Relation Alignment (CRA), to eliminate spurious correlations and improve the causal consistency between question-answering and video temporal grounding. Our CRA involves three essential components: i) Gaussian Smoothing Grounding (GSG) module for estimating the time interval via cross-modal attention, which is de-noised by an adaptive Gaussian filter, ii) Cross-Modal Alignment (CMA) enhances the performance of weakly supervised VideoQG by leveraging bidirectional contrastive learning between estimated video segments and QA features, iii) Explicit Causal Intervention (ECI) module for multimodal deconfounding, which involves front-door intervention for vision and back-door intervention for language. Extensive experiments on two VideoQG datasets demonstrate the superiority of our CRA in discovering visually grounded content and achieving robust question reasoning. Codes are available at https://github.com/WissingChen/CRA-GQA.

arXiv:2503.06380

TI-JEPA: An Innovative Energy-based Joint Embedding Strategy for Text-Image Multimodal Systems

This paper focuses on multimodal alignment within the realm of Artificial Intelligence, particularly in text and image modalities. The semantic gap between the textual and visual modality poses a discrepancy problem towards the effectiveness of multi-modalities fusion. Therefore, we introduce Text-Image Joint Embedding Predictive Architecture (TI-JEPA), an innovative pre-training strategy that leverages energy-based model (EBM) framework to capture complex cross-modal relationships. TI-JEPA combines the flexibility of EBM in self-supervised learning to facilitate the compatibility between textual and visual elements. Through extensive experiments across multiple benchmarks, we demonstrate that TI-JEPA achieves state-of-the-art performance on multimodal sentiment analysis task (and potentially on a wide range of multimodal-based tasks, such as Visual Question Answering), outperforming existing pre-training methodologies. Our findings highlight the potential of using energy-based framework in advancing multimodal fusion and suggest significant improvements for downstream applications.

arXiv:2611.21644v1cs.CL

MInfoVisQA: 정보 밀집형 시각 질의응답을 위한 다국어 벤치마크

MInfoVisQA: A Multilingual Benchmark for Information-Rich Visual Question Answering

멀티모달 대규모 언어 모델(MLLM)은 비전-언어 과제에서 뛰어난 성능을 보이지만, 표면 언어가 바뀔 때 정보 밀집형 시각 자료(InfoVis)에 대한 이해를 일관되게 유지하는지는 여전히 불분명하다. 이 능력을 정량화하기 위해 24개 언어에 걸친 128개의 시드 인스턴스에서 파생된 8,960개의 QA 구성으로 이루어진 병렬 벤치마크 MInfoVisQA를 소개한다. 수치·구조 데이터를 텍스트와 분리하고 번역된 레이블을 프로그램적으로 동기화하여, 시드가 일치하고 언어 간 병렬인 시각 자료와 QA 쌍을 구축한다. 25개의 MLLM을 두 가지 설정, 즉 언어 정렬 QA(LQA, 시각 자료와 텍스트의 언어가 일치)와 교차 언어 QA(XQA, QA 언어는 고정하고 시각 자료의 언어는 변화)에서 평가한다. 결과는 종합 정확도가 상당한 강건성 격차를 가린다는 점을 보여 준다. 평균 LQA 정확도가 비슷한(80.4\%) 두 모델이 언어 간에 각각 7.8 및 20.3 퍼센트 포인트의 성능 편차를 보였다. 또한 영어 QA를 고정한 상태에서 25개 모델 모두 비영어 시각 자료에서 더 낮은 점수를 기록하였으며, 이는 교차 언어 민감도 지표의 필요성을 부각한다.

arXiv:2605.31572v2cs.CV

nuReasoning: 롱테일 자율주행을 위한 추론 중심 데이터셋 및 벤치마크

nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving

추론은 차량이 상식 지식을 적용하고, 공간 관계를 이해하고, 에이전트 간 상호작용을 추론하며, 안전한 결정을 내려야 하는 롱테일 시나리오에서 자율주행(AD)에 필수적이다. 그러나 기존 AD 데이터셋과 벤치마크는 주로 인지, 예측 또는 계획을 대상으로 하며, 현실적인 롱테일 주행 장면에 대한 추론에는 제한적인 지도만을 제공한다. 우리는 추론 중심 AD를 위한 대규모 실제 데이터셋 및 벤치마크인 nuReasoning을 소개한다. nuScenes와 nuPlan의 계보를 따라, nuReasoning은 실제 AD 데이터셋과 벤치마크를 롱테일 주행 시나리오에서의 추론 방향으로 발전시킨다. 이 데이터셋은 여러 도시에서 수집된 각 20초 길이의 20,000개 클립을 포함하며, 동기화된 다중 카메라 이미지, LiDAR 데이터, HD 맵, 객체 주석, 그리고 공간 추론, 의사결정 추론, 반사실적 추론에 걸친 사람이 검증한 추론 주석을 갖춘다. 주로 시각 질의응답에 초점을 맞춘 기존 데이터셋과 달리, nuReasoning은 추론 평가와 계획 평가를 모두 지원하여 추론 지도가 주행 성능에 미치는 영향을 직접 연구할 수 있게 한다. 실험 결과, nuReasoning으로 VLM을 미세 조정하면 주행 특화 질의응답이 크게 향상되며, VLA 학습에 추론 지도를 포함하면 추론 시 텍스트 추론 출력을 비활성화하더라도 계획 성능이 향상된다. 이러한 결과는 nuReasoning이 현실적인 롱테일 환경에서 강건하고 해석 가능한 추론 기반 AD 시스템을 평가하고 개선하기 위한 기반임을 보여 준다.

arXiv:2003.12511

Assessing Image Quality Issues for Real-World Problems

We introduce a new large-scale dataset that links the assessment of image quality issues to two practical vision tasks: image captioning and visual question answering. First, we identify for 39,181 images taken by people who are blind whether each is sufficient quality to recognize the content as well as what quality flaws are observed from six options. These labels serve as a critical foundation for us to make the following contributions: (1) a new problem and algorithms for deciding whether an image is insufficient quality to recognize the content and so not captionable, (2) a new problem and algorithms for deciding which of six quality flaws an image contains, (3) a new problem and algorithms for deciding whether a visual question is unanswerable due to unrecognizable content versus the content of interest being missing from the field of view, and (4) a novel application of more efficiently creating a large-scale image captioning dataset by automatically deciding whether an image is insufficient quality and so should not be captioned. We publicly-share our datasets and code to facilitate future extensions of this work: https://vizwiz.org.

arXiv:2510.07567v2cs.CV

비전-언어 모델에서의 교차 모달 어텐션 기반 언러닝

Cross-Modal Attention Guided Unlearning in Vision-Language Models

비전-언어 모델(VLM)은 시각적 맥락과 텍스트 맥락을 동시에 바탕으로 출력을 추론해야 하는 시각 질의응답(VQA) 같은 과제에서 멀티모달 이해 능력을 뛰어나게 입증해 왔다. 그러나 모델은 학습 중 일부 개인 정보 및/또는 민감 정보를 기억했다가 추론 시 그대로 내놓을 수 있다. 최근 LLM에서 이러한 개인 데이터 유출을 해결하기 위해 머신 언러닝이 활용되고 있다. VLM에서는 질의의 시각적 맥락에도 민감한 정보가 포함될 수 있어 이 과정이 한층 복잡해진다. 이 문제를 해결하기 위해 우리는 VLM, 특히 VQA 과제를 위한 언러닝을 탐구한다. 우리는 교차 모달 어텐션을 사용하여 VLM의 출력 생성에서 시각 토큰이 하는 역할을 분석하고, 이를 활용해 가볍고 효율적인 VLM 언러닝 프레임워크인 교차 모달 어텐션 기반 언러닝(Cross-Modal Attention Guided Unlearning, CAGUL)을 제안한다. 계산 비용이 큰 파인튜닝 방법과 달리 CAGUL은 외부 모듈을 사용하여 해당 질의에서 중요도가 낮은 시각 토큰에 언러닝 정보를 인코딩한다. 실험 결과, 변환된 시각 토큰은 사전 학습된 모델의 파라미터를 변경하지 않고도 유출을 방지하고 기준 모델의 동작을 유지하여, 망각 품질과 모델 유용성 사이에서 바람직한 절충을 달성한다.

arXiv:2308.16349

Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning Based on Visually Grounded Conversations

We introduce Affective Visual Dialog, an emotion explanation and reasoning task as a testbed for research on understanding the formation of emotions in visually grounded conversations. The task involves three skills: (1) Dialog-based Question Answering (2) Dialog-based Emotion Prediction and (3) Affective emotion explanation generation based on the dialog. Our key contribution is the collection of a large-scale dataset, dubbed AffectVisDial, consisting of 50K 10-turn visually grounded dialogs as well as concluding emotion attributions and dialog-informed textual emotion explanations, resulting in a total of 27,180 working hours. We explain our design decisions in collecting the dataset and introduce the questioner and answerer tasks that are associated with the participants in the conversation. We train and demonstrate solid Affective Visual Dialog baselines adapted from state-of-the-art models. Remarkably, the responses generated by our models show promising emotional reasoning abilities in response to visually grounded conversations. Our project page is available at https://affective-visual-dialog.github.io.