PyTorchKR Papers

“Visual Question Answering”

총 2,807편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2402.05374

CIC: A Framework for Culturally-Aware Image Captioning

Image Captioning generates descriptive sentences from images using Vision-Language Pre-trained models (VLPs) such as BLIP, which has improved greatly. However, current methods lack the generation of detailed descriptive captions for the cultural elements depicted in the images, such as the traditional clothing worn by people from Asian cultural groups. In this paper, we propose a new framework, Culturally-aware Image Captioning (CIC), that generates captions and describes cultural elements extracted from cultural visual elements in images representing cultures. Inspired by methods combining visual modality and Large Language Models (LLMs) through appropriate prompts, our framework (1) generates questions based on cultural categories from images, (2) extracts cultural visual elements from Visual Question Answering (VQA) using generated questions, and (3) generates culturally-aware captions using LLMs with the prompts. Our human evaluation conducted on 45 participants from 4 different cultural groups with a high understanding of the corresponding culture shows that our proposed framework generates more culturally descriptive captions when compared to the image captioning baseline based on VLPs. Resources can be found at https://shane3606.github.io/cic..

arXiv:2010.16010공식 코드 · Papers with Code

Loss re-scaling VQA: Revisiting the LanguagePrior Problem from a Class-imbalance View

Recent studies have pointed out that many well-developed Visual Question Answering (VQA) models are heavily affected by the language prior problem, which refers to making predictions based on the co-occurrence pattern between textual questions and answers instead of reasoning visual contents. To tackle it, most existing methods focus on enhancing visual feature learning to reduce this superficial textual shortcut influence on VQA model decisions. However, limited effort has been devoted to providing an explicit interpretation for its inherent cause. It thus lacks a good guidance for the research community to move forward in a purposeful way, resulting in model construction perplexity in overcoming this non-trivial problem. In this paper, we propose to interpret the language prior problem in VQA from a class-imbalance view. Concretely, we design a novel interpretation scheme whereby the loss of mis-predicted frequent and sparse answers of the same question type is distinctly exhibited during the late training phase. It explicitly reveals why the VQA model tends to produce a frequent yet obviously wrong answer, to a given question whose right answer is sparse in the training set. Based upon this observation, we further develop a novel loss re-scaling approach to assign different weights to each answer based on the training data statistics for computing the final loss. We apply our approach into three baselines and the experimental results on two VQA-CP benchmark datasets evidently demonstrate its effectiveness. In addition, we also justify the validity of the class imbalance interpretation scheme on other computer vision tasks, such as face recognition and image classification.

arXiv:1712.01238

Learning by Asking Questions

We introduce an interactive learning framework for the development and testing of intelligent visual systems, called learning-by-asking (LBA). We explore LBA in context of the Visual Question Answering (VQA) task. LBA differs from standard VQA training in that most questions are not observed during training time, and the learner must ask questions it wants answers to. Thus, LBA more closely mimics natural learning and has the potential to be more data-efficient than the traditional VQA setting. We present a model that performs LBA on the CLEVR dataset, and show that it automatically discovers an easy-to-hard curriculum when learning interactively from an oracle. Our LBA generated data consistently matches or outperforms the CLEVR train data and is more sample efficient. We also show that our model asks questions that generalize to state-of-the-art VQA models and to novel test time distributions.

arXiv:2503.15035

GraspCorrect: Robotic Grasp Correction via Vision-Language Model-Guided Feedback

Despite significant advancements in robotic manipulation, achieving consistent and stable grasping remains a fundamental challenge, often limiting the successful execution of complex tasks. Our analysis reveals that even state-of-the-art policy models frequently exhibit unstable grasping behaviors, leading to failure cases that create bottlenecks in real-world robotic applications. To address these challenges, we introduce GraspCorrect, a plug-and-play module designed to enhance grasp performance through vision-language model-guided feedback. GraspCorrect employs an iterative visual question-answering framework with two key components: grasp-guided prompting, which incorporates task-specific constraints, and object-aware sampling, which ensures the selection of physically feasible grasp candidates. By iteratively generating intermediate visual goals and translating them into joint-level actions, GraspCorrect significantly improves grasp stability and consistently enhances task success rates across existing policy models in the RLBench and CALVIN datasets.

arXiv:2501.02964공식 코드 · Papers with Code

Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild

Complex visual reasoning remains a key challenge today. Typically, the challenge is tackled using methodologies such as Chain of Thought (COT) and visual instruction tuning. However, how to organically combine these two methodologies for greater success remains unexplored. Also, issues like hallucinations and high training cost still need to be addressed. In this work, we devise an innovative multi-round training and reasoning framework suitable for lightweight Multimodal Large Language Models (MLLMs). Our self-questioning approach heuristically guides MLLMs to focus on visual clues relevant to the target problem, reducing hallucinations and enhancing the model's ability to describe fine-grained image details. This ultimately enables the model to perform well in complex visual reasoning and question-answering tasks. We have named this framework Socratic Questioning(SQ). To facilitate future research, we create a multimodal mini-dataset named CapQA, which includes 1k images of fine-grained activities, for visual instruction tuning and evaluation, our proposed SQ method leads to a 31.2% improvement in the hallucination score. Our extensive experiments on various benchmarks demonstrate SQ's remarkable capabilities in heuristic self-questioning, zero-shot visual reasoning and hallucination mitigation. Our model and code will be publicly available.

arXiv:2611.16517v1cs.CV

UniSpark: 멀티모달 이해와 생성을 위한 하이브리드 어텐션 탐구

UniSpark: Exploring Hybrid Attention for Multimodal Understanding and Generation

본 논문은 시각 이해와 생성을 위한 하이브리드 어텐션을 연구하는 통합 멀티모달 모델 UniSpark-9B를 제안한다. UniSpark는 각 브랜치에 약 9B 파라미터를 갖는 Mixture-of-Transformer-Experts 백본을 기반으로 하며, 명시적인 토큰 간 상호작용을 위한 전체 어텐션과 간결한 문맥 전파를 위한 GatedDeltaNet 선형 어텐션을 결합한다. 자기회귀 텍스트 예측과 시각 플로 매칭은 시각 질의응답, 텍스트-이미지 생성, 지시 기반 편집, 텍스트-이미지 교차 예측을 함께 지원한다. 교차 생성에서는 노이즈가 포함된 시각 목표를 이후 예측으로부터 분리하고, 깨끗한 시각 표현은 이후 출력을 위한 문맥으로 제공한다. 블록 단위 프로토콜은 전체 어텐션 계층에서는 어텐션 마스크를, 선형 어텐션 계층에서는 순환 상태 유지 규칙을 통해 이러한 구분을 강제한다. 선별된 학습 데이터에는 159.7M개의 생성 샘플, 213.4M개의 이해 샘플, 624.8K개의 교차 시퀀스가 포함된다. 4단계 커리큘럼은 멀티모달 이해와 시각 생성을 함께 학습하며, 답변 정확도와 지시 이행을 개선하는 동시에 이미지 합성을 2K 해상도까지 확장한다. UniSpark는 MME-S에서 2470.3점, GenEval에서 0.83점을 기록한다. 정성적 사례는 교차 이미지 생성, 비디오 합성, 비디오 편집에서의 능력을 추가로 보여준다. 이러한 결과는 하이브리드 어텐션이 통합 멀티모달 이해와 생성을 위한 효과적인 백본이 됨을 입증한다.

arXiv:1906.09954

Integrating Knowledge and Reasoning in Image Understanding

Deep learning based data-driven approaches have been successfully applied in various image understanding applications ranging from object recognition, semantic segmentation to visual question answering. However, the lack of knowledge integration as well as higher-level reasoning capabilities with the methods still pose a hindrance. In this work, we present a brief survey of a few representative reasoning mechanisms, knowledge integration methods and their corresponding image understanding applications developed by various groups of researchers, approaching the problem from a variety of angles. Furthermore, we discuss upon key efforts on integrating external knowledge with neural networks. Taking cues from these efforts, we conclude by discussing potential pathways to improve reasoning capabilities.

arXiv:2410.08209

Emerging Pixel Grounding in Large Multimodal Models Without Grounding Supervision

Current large multimodal models (LMMs) face challenges in grounding, which requires the model to relate language components to visual entities. Contrary to the common practice that fine-tunes LMMs with additional grounding supervision, we find that the grounding ability can in fact emerge in LMMs trained without explicit grounding supervision. To reveal this emerging grounding, we introduce an "attend-and-segment" method which leverages attention maps from standard LMMs to perform pixel-level segmentation. Furthermore, to enhance the grounding ability, we propose DIFFLMM, an LMM utilizing a diffusion-based visual encoder, as opposed to the standard CLIP visual encoder, and trained with the same weak supervision. Without being constrained by the biases and limited scale of grounding-specific supervision data, our approach is more generalizable and scalable. We achieve competitive performance on both grounding-specific and general visual question answering benchmarks, compared with grounding LMMs and generalist LMMs, respectively. Notably, we achieve a 44.2 grounding mask recall on grounded conversation generation without any grounding supervision, outperforming the extensively supervised model GLaMM. Project page: https://groundLMM.github.io.

arXiv:2302.13668공식 코드 · Papers with Code

Contrastive Video Question Answering via Video Graph Transformer

We propose to perform video question answering (VideoQA) in a Contrastive manner via a Video Graph Transformer model (CoVGT). CoVGT's uniqueness and superiority are three-fold: 1) It proposes a dynamic graph transformer module which encodes video by explicitly capturing the visual objects, their relations and dynamics, for complex spatio-temporal reasoning. 2) It designs separate video and text transformers for contrastive learning between the video and text to perform QA, instead of multi-modal transformer for answer classification. Fine-grained video-text communication is done by additional cross-modal interaction modules. 3) It is optimized by the joint fully- and self-supervised contrastive objectives between the correct and incorrect answers, as well as the relevant and irrelevant questions respectively. With superior video encoding and QA solution, we show that CoVGT can achieve much better performances than previous arts on video reasoning tasks. Its performances even surpass those models that are pretrained with millions of external data. We further show that CoVGT can also benefit from cross-modal pretraining, yet with orders of magnitude smaller data. The results demonstrate the effectiveness and superiority of CoVGT, and additionally reveal its potential for more data-efficient pretraining. We hope our success can advance VideoQA beyond coarse recognition/description towards fine-grained relation reasoning of video contents. Our code is available at https://github.com/doc-doc/CoVGT.

arXiv:2611.17900v1cs.CV

AmbiSteer: 비전-언어 모델에서 모호성 인식 응답을 선택적으로 유도하기

AmbiSteer: Selectively Eliciting Ambiguity-Aware Responses in Vision-Language Models

비전-언어 모델(VLM)은 시각적 증거가 불충분한 경우에도 단정적인 답변을 제시하는 과신 경향을 자주 보인다. 대표적인 사례는 시각 질의응답(VQA)에서의 지시 대상 모호성으로, 이미지 속 여러 개체가 질문의 지시 대상이 될 수 있는 경우이다. 이러한 사례를 다루려면 모호성이 존재하는지를 인식해야 한다. 모호한 질문은 명확화를 요청하거나 가능한 지시 대상을 열거하여 처리할 수 있으며, 모호하지 않은 질문에는 직접 답해야 한다. 현재의 VLM은 이러한 구분에 어려움을 겪으며 흔히 하나의 해석으로 기울어진다. 활성화 조향(activation steering)이 이러한 행동을 유도하는 추론 시점 메커니즘으로 활용될 수 있는지 조사하기 위해, 본 연구는 모호성 인식 행동을 선택적으로 유도하는 경량 프레임워크인 AmbiSteer를 제안한다. 세 가지 VLM에 걸쳐 조향만으로는 효과가 제한적이며 모델에 따라 달라지는 반면, 프롬프팅이 일반적으로 더 효과적임을 확인하였다. 프롬프팅과 조향을 결합하면 명확화 및 열거 행동이 크게 증가하지만, 모호하지 않은 사례에서도 이러한 행동이 유도된다. AmbiSteer는 선형 분류기로 입력의 모호성을 예측하고 이에 따라 조향 방향을 조건화하여 이 문제를 해결하며, 적절하게 처리되는 입력의 비율을 모델 평균 59.0%에서 77.0%로 높인다. 본 연구는 불확실한 멀티모달 환경에서 모델의 과신을 줄이는 경량 접근법의 토대를 마련한다.

arXiv:1904.06038

Evaluating the Representational Hub of Language and Vision Models

The multimodal models used in the emerging field at the intersection of computational linguistics and computer vision implement the bottom-up processing of the `Hub and Spoke' architecture proposed in cognitive science to represent how the brain processes and combines multi-sensory inputs. In particular, the Hub is implemented as a neural network encoder. We investigate the effect on this encoder of various vision-and-language tasks proposed in the literature: visual question answering, visual reference resolution, and visually grounded dialogue. To measure the quality of the representations learned by the encoder, we use two kinds of analyses. First, we evaluate the encoder pre-trained on the different vision-and-language tasks on an existing diagnostic task designed to assess multimodal semantic understanding. Second, we carry out a battery of analyses aimed at studying how the encoder merges and exploits the two modalities.

arXiv:2505.14462공식 코드 · Papers with Code

RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding

As vision-language models (VLMs) become increasingly integrated into daily life, the need for accurate visual culture understanding is becoming critical. Yet, these models frequently fall short in interpreting cultural nuances effectively. Prior work has demonstrated the effectiveness of retrieval-augmented generation (RAG) in enhancing cultural understanding in text-only settings, while its application in multimodal scenarios remains underexplored. To bridge this gap, we introduce RAVENEA (Retrieval-Augmented Visual culturE uNdErstAnding), a new benchmark designed to advance visual culture understanding through retrieval, focusing on two tasks: culture-focused visual question answering (cVQA) and culture-informed image captioning (cIC). RAVENEA extends existing datasets by integrating over 10,000 Wikipedia documents curated and ranked by human annotators. With RAVENEA, we train and evaluate seven multimodal retrievers for each image query, and measure the downstream impact of retrieval-augmented inputs across fourteen state-of-the-art VLMs. Our results show that lightweight VLMs, when augmented with culture-aware retrieval, outperform their non-augmented counterparts (by at least 3.2% absolute on cVQA and 6.2% absolute on cIC). This highlights the value of retrieval-augmented methods and culturally inclusive benchmarks for multimodal understanding.

arXiv:2307.04192공식 코드 · Papers with Code

Self-Adaptive Sampling for Efficient Video Question-Answering on Image--Text Models

Video question-answering is a fundamental task in the field of video understanding. Although current vision--language models (VLMs) equipped with Video Transformers have enabled temporal modeling and yielded superior results, they are at the cost of huge computational power and thus too expensive to deploy in real-time application scenarios. An economical workaround only samples a small portion of frames to represent the main content of that video and tune an image--text model on these sampled frames. Recent video understanding models usually randomly sample a set of frames or clips, regardless of internal correlations between their visual contents, nor their relevance to the problem. We argue that such kinds of aimless sampling may omit the key frames from which the correct answer can be deduced, and the situation gets worse when the sampling sparsity increases, which always happens as the video lengths increase. To mitigate this issue, we propose two frame sampling strategies, namely the most domain frames (MDF) and most implied frames (MIF), to maximally preserve those frames that are most likely vital to the given questions. MDF passively minimizes the risk of key frame omission in a bootstrap manner, while MIS actively searches key frames customized for each video--question pair with the assistance of auxiliary models. The experimental results on three public datasets from three advanced VLMs (CLIP, GIT and All-in-one) demonstrate that our proposed strategies can boost the performance for image-text pretrained models. The source codes pertaining to the method proposed in this paper are publicly available at https://github.com/declare-lab/sas-vqa.

arXiv:2205.11686

On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization

Combining the visual modality with pretrained language models has been surprisingly effective for simple descriptive tasks such as image captioning. More general text generation however remains elusive. We take a step back and ask: How do these models work for more complex generative tasks, i.e. conditioning on both text and images? Are multimodal models simply visually adapted language models, or do they combine they reason jointly over modalities? We investigate these questions in the context of self-rationalization (jointly generating task labels/answers and free-text explanations) of three tasks: (i) visual question answering in VQA-X, (ii) visual commonsense reasoning in VCR, and (iii) visual-textual entailment in e-SNLI-VE. We show that recent unimodal advances, CLIP image representations and scaling of language models, do not consistently improve self-rationalization in multimodal tasks. We find that no single model type works universally best across tasks, datasets, and finetuning data sizes. Our findings motivate the need for novel general backbones approach that move text generation from images and text beyond image captioning.

arXiv:2405.01474공식 코드 · Papers with Code

Understanding Figurative Meaning through Explainable Visual Entailment

Large Vision-Language Models (VLMs) have demonstrated strong capabilities in tasks requiring a fine-grained understanding of literal meaning in images and text, such as visual question-answering or visual entailment. However, there has been little exploration of these models' capabilities when presented with images and captions containing figurative meaning, such as metaphors or humor. To close this gap, we propose a new task framing the figurative meaning understanding problem as an explainable visual entailment task, where the model has to predict whether the image (premise) entails a caption (hypothesis) and justify the predicted label with a textual explanation. The figurative phenomena can be present either in the image, the caption, or both. Utilizing a human-AI collaboration approach, we build the accompanying expert-verified dataset V-FLUTE, containing 6,027 {image, caption, label, explanation} instances spanning five diverse figurative phenomena: metaphors, similes, idioms, sarcasm, and humor. Through automatic evaluation, we find that VLMs struggle to generalize from literal to figurative meaning, particularly when it is present in images. Further, we identify common types of errors in VLM reasoning via human evaluation.

arXiv:2609.23601v1cs.CV

PREM: 장시간 비디오 이해를 위한 접두 문맥 기반 순환 메모리

PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding

장시간 비디오 이해(long-video understanding)는 엄격한 토큰(token) 예산 하에서 일시적인 시각적 증거를 포착해야 하지만, 기존 방법들은 프레임을 압축하거나, 메모리 토큰을 추가하거나, 내부 키-값(KV) 캐시를 변경한다. 본 논문에서는 고정된(frozen) 비전-언어 모델(VLM)을 위한 메모리 토큰이 필요 없는 프레임워크인 접두 문맥 기반 순환 메모리(Prefix-Steered Recurrent Memory, PREM)를 제안한다. PREM은 비디오 수집(ingestion)과 질의 응답을 분리한다: 순환 작성기(recurrent writer)가 시각적 스트림을 256 KiB 크기의 컴팩트한 다중 슬롯 연상 상태(multi-slot associative state)로 증류(distill)하는 동안, 질문 조건부 판독기(question-conditioned readout)는 프리필(prefill) 과정에서 메모리에서 유래한 키/값(K/V) 조향(steering) 변조를 기존의 비시각적 프롬프트 접두 문맥에 추가한다. 이를 통해 추가적인 프롬프트 토큰이나 디코딩 순환(recurrence) 없이 한 번 작성하고 여러 번 질의하는(write-once, query-many) 추론이 가능해진다. 오프라인 및 스트리밍 종료 시점(end-of-stream) 설정에서 6개의 장시간 비디오 벤치마크(benchmark)에 걸쳐, PREM은 평가된 모든 시각 예산 수준에서 고정(frozen) 베이스라인을 일관되게 능가한다. 16프레임으로 제한된 예산 하에서, PREM은 Qwen2.5-VL-3B에서 매크로 평균 정확도를 3.06% 향상시키며, 행동 반의어 식별(action antonym identification)에서 11.0%, 국소화된 니들 검색(localized needle retrieval)에서 9.9%의 향상을 보인다. 이러한 향상은 백본(backbone) 파라미터의 0.24%만을 튜닝하고, 최대 GPU 메모리 오버헤드 0.03 GiB만을 필요로 한다.

arXiv:2210.11253

Image Semantic Relation Generation

Scene graphs provide structured semantic understanding beyond images. For downstream tasks, such as image retrieval, visual question answering, visual relationship detection, and even autonomous vehicle technology, scene graphs can not only distil complex image information but also correct the bias of visual models using semantic-level relations, which has broad application prospects. However, the heavy labour cost of constructing graph annotations may hinder the application of PSG in practical scenarios. Inspired by the observation that people usually identify the subject and object first and then determine the relationship between them, we proposed to decouple the scene graphs generation task into two sub-tasks: 1) an image segmentation task to pick up the qualified objects. 2) a restricted auto-regressive text generation task to generate the relation between given objects. Therefore, in this work, we introduce image semantic relation generation (ISRG), a simple but effective image-to-text model, which achieved 31 points on the OpenPSG dataset and outperforms strong baselines respectively by 16 points (ResNet-50) and 5 points (CLIP).

arXiv:2212.09898

MetaCLUE: Towards Comprehensive Visual Metaphors Research

Creativity is an indispensable part of human cognition and also an inherent part of how we make sense of the world. Metaphorical abstraction is fundamental in communicating creative ideas through nuanced relationships between abstract concepts such as feelings. While computer vision benchmarks and approaches predominantly focus on understanding and generating literal interpretations of images, metaphorical comprehension of images remains relatively unexplored. Towards this goal, we introduce MetaCLUE, a set of vision tasks on visual metaphor. We also collect high-quality and rich metaphor annotations (abstract objects, concepts, relationships along with their corresponding object boxes) as there do not exist any datasets that facilitate the evaluation of these tasks. We perform a comprehensive analysis of state-of-the-art models in vision and language based on our annotations, highlighting strengths and weaknesses of current approaches in visual metaphor Classification, Localization, Understanding (retrieval, question answering, captioning) and gEneration (text-to-image synthesis) tasks. We hope this work provides a concrete step towards developing AI systems with human-like creative capabilities.

arXiv:2311.17043공식 코드 · Papers with Code

LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

In this work, we present a novel method to tackle the token generation challenge in Vision Language Models (VLMs) for video and image understanding, called LLaMA-VID. Current VLMs, while proficient in tasks like image captioning and visual question answering, face computational burdens when processing long videos due to the excessive visual tokens. LLaMA-VID addresses this issue by representing each frame with two distinct tokens, namely context token and content token. The context token encodes the overall image context based on user input, whereas the content token encapsulates visual cues in each frame. This dual-token strategy significantly reduces the overload of long videos while preserving critical information. Generally, LLaMA-VID empowers existing frameworks to support hour-long videos and pushes their upper limit with an extra context token. It is proved to surpass previous methods on most of video- or image-based benchmarks. Code is available https://github.com/dvlab-research/LLaMA-VID}{https://github.com/dvlab-research/LLaMA-VID

arXiv:1904.12584공식 코드 · Papers with Code

The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval.