PyTorchKR Papers

“Visual Question Answering”

총 2,807편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2406.19736공식 코드 · Papers with Code

MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment

This paper introduces MM-Instruct, a large-scale dataset of diverse and high-quality visual instruction data designed to enhance the instruction-following capabilities of large multimodal models (LMMs). While existing visual instruction datasets often focus on question-answering, they struggle to generalize to broader application scenarios such as creative writing, summarization, or image analysis. To address these limitations, we propose a novel approach to constructing MM-Instruct that leverages the strong instruction-following capabilities of existing LLMs to generate novel visual instruction data from large-scale but conventional image captioning datasets. MM-Instruct first leverages ChatGPT to automatically generate diverse instructions from a small set of seed instructions through augmenting and summarization. It then matches these instructions with images and uses an open-sourced large language model (LLM) to generate coherent answers to the instruction-image pairs. The LLM is grounded by the detailed text descriptions of images in the whole answer generation process to guarantee the alignment of the instruction data. Moreover, we introduce a benchmark based on the generated instruction data to evaluate the instruction-following capabilities of existing LMMs. We demonstrate the effectiveness of MM-Instruct by training a LLaVA-1.5 model on the generated data, denoted as LLaVA-Instruct, which exhibits significant improvements in instruction-following capabilities compared to LLaVA-1.5 models. The MM-Instruct dataset, benchmark, and pre-trained models are available at https://github.com/jihaonew/MM-Instruct.

arXiv:2607.14660v1cs.CV

VIABench: 시각장애인으로부터 수집한 시각장애 보조를 위한 포괄적 비디오 벤치마크

VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance

시각장애인(visually impaired individuals, VII)은 시각 정보에 대한 접근이 제한되어 일상에서 상당한 어려움을 겪는다. 다중모달 대규모 언어 모델(Multimodal Large Language Models, MLLM)은 일반적인 비전 및 언어 과제에서 인상적인 성과를 거두었지만, 실세계의 시각장애 보조에서의 실용적 유용성은 여전히 크게 미탐색 상태로 남아 있다. 이 공백을 메우기 위해, 우리는 시각장애인(VII)이 직접 녹화하거나 공유한 1인칭 비디오를 사용하여 시각장애 보조(Visually Impaired Assistance) 시나리오에서 MLLM을 평가하도록 특별히 설계된 포괄적 비디오 벤치마크인 VIABench를 소개한다. VIABench는 시각 보조에서 서로 다른 요구를 겨냥하는 세 가지 핵심 과제를 정의한다. 선제적 알림(Proactive Reminder): 진행 중인 비디오 내용을 해석하면서 다가오는 내비게이션상 중요한 사건을 선제적으로 예측하고 구두로 설명하는 모델의 능력을 평가한다; 시각 질의응답(Visual Question Answering, VQA): 비디오 내 환경 또는 사물에 대한 사용자 질문에 답하는 모델의 역량을 평가한다; 비전 기반 상호작용(Vision-Guided Interaction): 사용자와 환경 사이의 의도적 상호작용을 수행하기 위한 맥락 인식 추론을 검증한다. 견고하고 공정한 평가를 위해, 온라인(실시간) 및 오프라인 설정을 모두 지원하는 엄격한 벤치마킹 파이프라인을 제안한다. 우리의 실험은 현재의 MLLM이 여전히 VII을 위한 포괄적 지원을 제공하는 데 어려움을 겪음을 보여주며, 특히 정확한 예측과 실시간 반응성을 요구하는 선제적 알림 과제에서 그렇다. 우리는 VIABench가 실세계 보조를 위한 맞춤형 MLLM 개발을 향한 향후 연구를 이끌어, 궁극적으로 시각장애인의 내비게이션 및 상호작용 경험을 개선하기를 기대한다. 코드와 데이터는 https://github.com/MCG-NJU/VIABench에서 공개될 예정이다.

arXiv:2104.03046

Multimodal Continuous Visual Attention Mechanisms

Visual attention mechanisms are a key component of neural network models for computer vision. By focusing on a discrete set of objects or image regions, these mechanisms identify the most relevant features and use them to build more powerful representations. Recently, continuous-domain alternatives to discrete attention models have been proposed, which exploit the continuity of images. These approaches model attention as simple unimodal densities (e.g. a Gaussian), making them less suitable to deal with images whose region of interest has a complex shape or is composed of multiple non-contiguous patches. In this paper, we introduce a new continuous attention mechanism that produces multimodal densities, in the form of mixtures of Gaussians. We use the EM algorithm to obtain a clustering of relevant regions in the image, and a description length penalty to select the number of components in the mixture. Our densities decompose as a linear combination of unimodal attention mechanisms, enabling closed-form Jacobians for the backpropagation step. Experiments on visual question answering in the VQA-v2 dataset show competitive accuracies and a selection of regions that mimics human attention more closely in VQA-HAT. We present several examples that suggest how multimodal attention maps are naturally more interpretable than their unimodal counterparts, showing the ability of our model to automatically segregate objects from ground in complex scenes.

arXiv:2607.29445v1cs.CV

적외선 비전-언어 모델에 대한 표적 의미론적 공격을 위한 QR 구조 열 트리거

QR-Structured Thermal Triggers for Targeted Semantic Attacks on Infrared Vision-Language Models

적외선 비전-언어 모델(IR-VLM)은 열 화상 인식을 개방형 어휘(open-vocabulary) 분류, 이미지 캡셔닝 및 시각적 질문 답변(VQA)으로 확장합니다. 그러나 구조화된 열 섭동에 대한 견고성과 모달 간 의미론적 정렬(cross-modal semantic alignment)의 안정성은 여전히 충분히 연구되지 않았습니다. 본 논문에서는 IR-VLM의 표적 의미론적 유도를 위한 은밀하고 학습이 필요 없는(training-free) 블랙박스 프레임워크인 QR 구조 열 트리거(QR-STT)를 제안합니다. QR-STT는 QR 패턴의 기능적 영역을 보존하면서 각 모듈에 저온, 중립 또는 고온의 열 상태가 할당되는 내부 모듈을 최적화합니다. 이 프레임워크는 위치, 크기, 회전, 강도, 블러 및 둥글기를 포함한 렌더링 파라미터와 모듈 위상을 공동으로 탐색합니다. 탐욕적 모듈 반전 정제를 포함하는 3단계 무그래디언트 절차는 혼합된 이산 및 연속 탐색 공간을 효율적으로 처리합니다. 목적 함수는 공격자가 선택한 목표와의 정렬을 촉진하고, 원본 클래스 증거를 억제하며, QR 구조와 시각적 유사성을 정규화합니다. 여러 CLIP 스타일 인코더에 대한 실험 결과, QR-STT는 시각적 은밀성을 유지하면서 이미지-텍스트 정렬을 선택된 개념으로 일관되게 재지정함을 보여줍니다. 분류에 최적화된 섭동은 이미지 캡셔닝 및 VQA로도 전이되어 생성된 출력에서 목표 일관적 의미 드리프트를 유발합니다. 이러한 결과는 QR 구조 열 패턴이 언어 기반 적외선 인식을 위한 해석 가능한 공격 표면임을 입증하며, 구조화된 교차 태스크 의미론적 공격에 대한 견고성 평가의 필요성을 강조합니다.

arXiv:2211.16636

Iterative Scene Graph Generation with Generative Transformers

Scene graphs provide a rich, structured representation of a scene by encoding the entities (objects) and their spatial relationships in a graphical format. This representation has proven useful in several tasks, such as question answering, captioning, and even object detection, to name a few. Current approaches take a generation-by-classification approach where the scene graph is generated through labeling of all possible edges between objects in a scene, which adds computational overhead to the approach. This work introduces a generative transformer-based approach to generating scene graphs beyond link prediction. Using two transformer-based components, we first sample a possible scene graph structure from detected objects and their visual features. We then perform predicate classification on the sampled edges to generate the final scene graph. This approach allows us to efficiently generate scene graphs from images with minimal inference overhead. Extensive experiments on the Visual Genome dataset demonstrate the efficiency of the proposed approach. Without bells and whistles, we obtain, on average, 20.7% mean recall (mR@100) across different settings for scene graph generation (SGG), outperforming state-of-the-art SGG approaches while offering competitive performance to unbiased SGG approaches.

arXiv:2407.20845

Assessing Graphical Perception of Image Embedding Models using Channel Effectiveness

Recent advancements in vision models have greatly improved their ability to handle complex chart understanding tasks, like chart captioning and question answering. However, it remains challenging to assess how these models process charts. Existing benchmarks only roughly evaluate model performance without evaluating the underlying mechanisms, such as how models extract image embeddings. This limits our understanding of the model's ability to perceive fundamental graphical components. To address this, we introduce a novel evaluation framework to assess the graphical perception of image embedding models. For chart comprehension, we examine two main aspects of channel effectiveness: accuracy and discriminability of various visual channels. Channel accuracy is assessed through the linearity of embeddings, measuring how well the perceived magnitude aligns with the size of the stimulus. Discriminability is evaluated based on the distances between embeddings, indicating their distinctness. Our experiments with the CLIP model show that it perceives channel accuracy differently from humans and shows unique discriminability in channels like length, tilt, and curvature. We aim to develop this work into a broader benchmark for reliable visual encoders, enhancing models for precise chart comprehension and human-like perception in future applications.

arXiv:2601.19151v2cs.AI

Multimodal Collaborative Debate for Zero-Shot Time Series Reasoning

Large language models (LLMs) are increasingly used as natural-language interfaces to structured data, yet they remain brittle when reasoning over time series. Visual patterns can be misleading, numerical claims can be hallucinated, and textual context can override evidence from the signal. We study zero-shot time-series reasoning as a multimodal evidence arbitration problem for LLM agents. We propose TS-Debate, an inference-time multi-agent protocol that requires no task-specific fine-tuning. TS-Debate first elicits relevant domain knowledge, then assigns modality-specialized agents to textual context, visual patterns, and numerical signals, and coordinates their interaction through a verification-conflict-calibration procedure. Reviewer agents check decision-critical claims with lightweight code execution and numerical lookup, resolve cross-modal disagreement, and calibrate the final answer. Unlike generic multi-agent debate or unconstrained tool use, TS-Debate specifies how evidence is exposed, which claims are checkable, and how verification outcomes shape synthesis. Across 20 tasks from three public benchmarks, TS-Debate improves classification and question answering performance over strong baselines, while revealing that debate is most useful for global-structure and cross-view reasoning rather than local value reconstruction.

arXiv:2004.14451공식 코드 · Papers with Code

Pragmatic Issue-Sensitive Image Captioning

Image captioning systems have recently improved dramatically, but they still tend to produce captions that are insensitive to the communicative goals that captions should meet. To address this, we propose Issue-Sensitive Image Captioning (ISIC). In ISIC, a captioning system is given a target image and an issue, which is a set of images partitioned in a way that specifies what information is relevant. The goal of the captioner is to produce a caption that resolves this issue. To model this task, we use an extension of the Rational Speech Acts model of pragmatic language use. Our extension is built on top of state-of-the-art pretrained neural image captioners and explicitly reasons about issues in our sense. We establish experimentally that these models generate captions that are both highly descriptive and issue-sensitive, and we show how ISIC can complement and enrich the related task of Visual Question Answering.

arXiv:2104.01394공식 코드 · Papers with Code

MMBERT: Multimodal BERT Pretraining for Improved Medical VQA

Images in the medical domain are fundamentally different from the general domain images. Consequently, it is infeasible to directly employ general domain Visual Question Answering (VQA) models for the medical domain. Additionally, medical images annotation is a costly and time-consuming process. To overcome these limitations, we propose a solution inspired by self-supervised pretraining of Transformer-style architectures for NLP, Vision and Language tasks. Our method involves learning richer medical image and text semantic representations using Masked Language Modeling (MLM) with image features as the pretext task on a large medical image+caption dataset. The proposed solution achieves new state-of-the-art performance on two VQA datasets for radiology images -- VQA-Med 2019 and VQA-RAD, outperforming even the ensemble models of previous best solutions. Moreover, our solution provides attention maps which help in model interpretability. The code is available at https://github.com/VirajBagal/MMBERT

arXiv:2201.00969

Interactive Attention AI to translate low light photos to captions for night scene understanding in women safety

There is amazing progress in Deep Learning based models for Image captioning and Low Light image enhancement. For the first time in literature, this paper develops a Deep Learning model that translates night scenes to sentences, opening new possibilities for AI applications in the safety of visually impaired women. Inspired by Image Captioning and Visual Question Answering, a novel Interactive Image Captioning is developed. A user can make the AI focus on any chosen person of interest by influencing the attention scoring. Attention context vectors are computed from CNN feature vectors and user-provided start word. The Encoder-Attention-Decoder neural network learns to produce captions from low brightness images. This paper demonstrates how women safety can be enabled by researching a novel AI capability in the Interactive Vision-Language model for perception of the environment in the night.

arXiv:2412.18185

TextMatch: Enhancing Image-Text Consistency Through Multimodal Optimization

Text-to-image generative models excel in creating images from text but struggle with ensuring alignment and consistency between outputs and prompts. This paper introduces TextMatch, a novel framework that leverages multimodal optimization to address image-text discrepancies in text-to-image (T2I) generation and editing. TextMatch employs a scoring strategy powered by large language models (LLMs) and visual question-answering (VQA) models to evaluate semantic consistency between prompts and generated images. By integrating multimodal in-context learning and chain of thought reasoning, our method dynamically refines prompts through iterative optimization. This process ensures that the generated images better capture user intent of, resulting in higher fidelity and relevance. Extensive experiments demonstrate that TextMatch significantly improves text-image consistency across multiple benchmarks, establishing a reliable framework for advancing the capabilities of text-to-image generative models. Our code is available at https://anonymous.4open.science/r/TextMatch-F55C/.

arXiv:2505.22897공식 코드 · Papers with Code

VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models

While bias in large language models (LLMs) is well-studied, similar concerns in vision-language models (VLMs) have received comparatively less attention. Existing VLM bias studies often focus on portrait-style images and gender-occupation associations, overlooking broader and more complex social stereotypes and their implied harm. This work introduces VIGNETTE, a large-scale VQA benchmark with 30M+ images for evaluating bias in VLMs through a question-answering framework spanning four directions: factuality, perception, stereotyping, and decision making. Beyond narrowly-centered studies, we assess how VLMs interpret identities in contextualized settings, revealing how models make trait and capability assumptions and exhibit patterns of discrimination. Drawing from social psychology, we examine how VLMs connect visual identity cues to trait and role-based inferences, encoding social hierarchies, through biased selections. Our findings uncover subtle, multifaceted, and surprising stereotypical patterns, offering insights into how VLMs construct social meaning from inputs.

arXiv:2411.17073공식 코드 · Papers with Code

Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answering

Accurate diagnosis and prognosis assisted by pathology images are essential for cancer treatment selection and planning. Despite the recent trend of adopting deep-learning approaches for analyzing complex pathology images, they fall short as they often overlook the domain-expert understanding of tissue structure and cell composition. In this work, we focus on a challenging Open-ended Pathology VQA (PathVQA-Open) task and propose a novel framework named Path-RAG, which leverages HistoCartography to retrieve relevant domain knowledge from pathology images and significantly improves performance on PathVQA-Open. Admitting the complexity of pathology image analysis, Path-RAG adopts a human-centered AI approach by retrieving domain knowledge using HistoCartography to select the relevant patches from pathology images. Our experiments suggest that domain guidance can significantly boost the accuracy of LLaVA-Med from 38% to 47%, with a notable gain of 28% for H&E-stained pathology images in the PathVQA-Open dataset. For longer-form question and answer pairs, our model consistently achieves significant improvements of 32.5% in ARCH-Open PubMed and 30.6% in ARCH-Open Books on H\&E images. Our code and dataset is available here (https://github.com/embedded-robotics/path-rag).

arXiv:2505.02278

Compositional Image-Text Matching and Retrieval by Grounding Entities

Vision-language pretraining on large datasets of images-text pairs is one of the main building blocks of current Vision-Language Models. While with additional training, these models excel in various downstream tasks, including visual question answering, image captioning, and visual commonsense reasoning. However, a notable weakness of pretrained models like CLIP, is their inability to perform entity grounding and compositional image and text matching~\cite{Jiang2024ComCLIP, yang2023amc, Rajabi2023GroundedVSR, learninglocalizeCVPR24}. In this work we propose a novel learning-free zero-shot augmentation of CLIP embeddings that has favorable compositional properties. We compute separate embeddings of sub-images of object entities and relations that are localized by the state of the art open vocabulary detectors and dynamically adjust the baseline global image embedding. % The final embedding is obtained by computing a weighted combination of the sub-image embeddings. The resulting embedding is then utilized for similarity computation with text embedding, resulting in a average 1.5\% improvement in image-text matching accuracy on the Visual Genome and SVO Probes datasets~\cite{krishna2017visualgenome, svo}. Notably, the enhanced embeddings demonstrate superior retrieval performance, thus achieving significant gains on the Flickr30K and MS-COCO retrieval benchmarks~\cite{flickr30ke, mscoco}, improving the state-of-the-art Recall@1 by 12\% and 0.4\%, respectively. Our code is available at https://github.com/madhukarreddyvongala/GroundingCLIP.

arXiv:2606.18992v1cs.CV

보여주고 묻지 마라: 턴 유효 커버리지를 갖는 합성 이미지 검색을 위한 생성적 시각적 모호성 해소

Show, Don't Ask: Generative Visual Disambiguation for Composed Image Retrieval with Turn-Valid Coverage

합성 이미지 검색(CIR)은 참조 이미지와 텍스트 수정을 사용하여 대상 이미지를 검색합니다. 그러나 이러한 쿼리는 종종 하나의 정확한 대상보다는 여러 가능한 이미지를 설명하므로, 사용자의 의도를 모호하게 만듭니다. 최근 방법들은 모호성을 추정하기 위해 컨포멀 예측(conformal prediction)을 사용하고 사용자에게 명확화 텍스트 질문을 함으로써 이 문제를 해결합니다. 그러나 이러한 방법에는 두 가지 제한이 있습니다: 커버리지 보장이 첫 번째 상호작용에서만 유지되며, 텍스트 질문은 외관, 속성 또는 시점과 같은 세밀한 시각적 차이를 해결하는 데 종종 부족합니다. 우리는 사용자에게 시각적 대안들의 작은 패널을 보여줌으로써 모호성을 해결하는 명확화 프레임워크인 CLARA를 제안합니다. 텍스트 질문에 답하는 대신, 사용자는 의도한 대상에 가장 가까운 프로토타입 이미지를 단순히 선택합니다. 이는 직접적인 시각적 신호를 제공하고 사용자의 답변을 예측하기 위해 모델(model)에 의존하는 것을 피합니다. 여러 상호작용 라운드에 걸쳐 유효한 컨포멀 보장을 유지하기 위해, CLARA는 사용자의 선택에 의해 유도된 가능도(likelihood) 비율을 사용하여 보정(calibration)을 재가중합니다. 표시된 프로토타입들은 또한 현재 후보 집합을 나타내도록 제약되며 실제 말뭉치(corpus) 이미지에 스냅(snapped)되어, 생성된 이미지가 커버리지를 인위적으로 향상시킬 수 없음을 보장합니다. 오픈 도메인 및 패션 벤치마크(benchmark)에 대한 실험 결과, CLARA는 단일 턴 최신 검색 성능과 대등하며, 상호작용 라운드 전반에 걸쳐 명목상 커버리지를 유지하고, 강력한 텍스트 질문 기준선(baseline)보다 더 적은 라운드로 의도한 대상을 찾는 것으로 나타났습니다. 시각적 명확화가 텍스트 질문보다 더 효과적인 시점이나 세밀한 속성과 관련된 모호성의 경우, 그 장점이 특히 명확합니다.

arXiv:2310.01960공식 코드 · Papers with Code

Language Models as Knowledge Bases for Visual Word Sense Disambiguation

Visual Word Sense Disambiguation (VWSD) is a novel challenging task that lies between linguistic sense disambiguation and fine-grained multimodal retrieval. The recent advancements in the development of visiolinguistic (VL) transformers suggest some off-the-self implementations with encouraging results, which however we argue that can be further improved. To this end, we propose some knowledge-enhancement techniques towards improving the retrieval performance of VL transformers via the usage of Large Language Models (LLMs) as Knowledge Bases. More specifically, knowledge stored in LLMs is retrieved with the help of appropriate prompts in a zero-shot manner, achieving performance advancements. Moreover, we convert VWSD to a purely textual question-answering (QA) problem by considering generated image captions as multiple-choice candidate answers. Zero-shot and few-shot prompting strategies are leveraged to explore the potential of such a transformation, while Chain-of-Thought (CoT) prompting in the zero-shot setting is able to reveal the internal reasoning steps an LLM follows to select the appropriate candidate. In total, our presented approach is the first one to analyze the merits of exploiting knowledge stored in LLMs in different ways to solve WVSD.

arXiv:2312.11666공식 코드 · Papers with Code

HAAR: Text-Conditioned Generative Model of 3D Strand-based Human Hairstyles

We present HAAR, a new strand-based generative model for 3D human hairstyles. Specifically, based on textual inputs, HAAR produces 3D hairstyles that could be used as production-level assets in modern computer graphics engines. Current AI-based generative models take advantage of powerful 2D priors to reconstruct 3D content in the form of point clouds, meshes, or volumetric functions. However, by using the 2D priors, they are intrinsically limited to only recovering the visual parts. Highly occluded hair structures can not be reconstructed with those methods, and they only model the ''outer shell'', which is not ready to be used in physics-based rendering or simulation pipelines. In contrast, we propose a first text-guided generative method that uses 3D hair strands as an underlying representation. Leveraging 2D visual question-answering (VQA) systems, we automatically annotate synthetic hair models that are generated from a small set of artist-created hairstyles. This allows us to train a latent diffusion model that operates in a common hairstyle UV space. In qualitative and quantitative studies, we demonstrate the capabilities of the proposed model and compare it to existing hairstyle generation approaches.

arXiv:2402.11684공식 코드 · Papers with Code

ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models

Large vision-language models (LVLMs) have shown premise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. However, they require considerable computational resources for training and deployment. This study aims to bridge the performance gap between traditional-scale LVLMs and resource-friendly lite versions by adopting high-quality training data. To this end, we propose a comprehensive pipeline for generating a synthetic dataset. The key idea is to leverage strong proprietary models to generate (i) fine-grained image annotations for vision-language alignment and (ii) complex reasoning visual question-answering pairs for visual instruction fine-tuning, yielding 1.3M samples in total. We train a series of lite VLMs on the synthetic dataset and experimental results demonstrate the effectiveness of the proposed scheme, where they achieve competitive performance on 17 benchmarks among 4B LVLMs, and even perform on par with 7B/13B-scale models on various benchmarks. This work highlights the feasibility of adopting high-quality data in crafting more efficient LVLMs. We name our dataset \textit{ALLaVA}, and open-source it to research community for developing better resource-efficient LVLMs for wider usage.

arXiv:2502.14420공식 코드 · Papers with Code

ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model

Humans possess a unified cognitive ability to perceive, comprehend, and interact with the physical world. Why can't large language models replicate this holistic understanding? Through a systematic analysis of existing training paradigms in vision-language-action models (VLA), we identify two key challenges: spurious forgetting, where robot training overwrites crucial visual-text alignments, and task interference, where competing control and understanding tasks degrade performance when trained jointly. To overcome these limitations, we propose ChatVLA, a novel framework featuring Phased Alignment Training, which incrementally integrates multimodal data after initial control mastery, and a Mixture-of-Experts architecture to minimize task interference. ChatVLA demonstrates competitive performance on visual question-answering datasets and significantly surpasses state-of-the-art vision-language-action (VLA) methods on multimodal understanding benchmarks. Notably, it achieves a six times higher performance on MMMU and scores 47.2% on MMStar with a more parameter-efficient design than ECoT. Furthermore, ChatVLA demonstrates superior performance on 25 real-world robot manipulation tasks compared to existing VLA methods like OpenVLA. Our findings highlight the potential of our unified framework for achieving both robust multimodal understanding and effective robot control.

arXiv:2502.18023

Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference

Despite the advancements made in Visual Large Language Models (VLLMs), like text Large Language Models (LLMs), they have limitations in addressing questions that require real-time information or are knowledge-intensive. Indiscriminately adopting Retrieval Augmented Generation (RAG) techniques is an effective yet expensive way to enable models to answer queries beyond their knowledge scopes. To mitigate the dependence on retrieval and simultaneously maintain, or even improve, the performance benefits provided by retrieval, we propose a method to detect the knowledge boundary of VLLMs, allowing for more efficient use of techniques like RAG. Specifically, we propose a method with two variants that fine-tunes a VLLM on an automatically constructed dataset for boundary identification. Experimental results on various types of Visual Question Answering datasets show that our method successfully depicts a VLLM's knowledge boundary based on which we are able to reduce indiscriminate retrieval while maintaining or improving the performance. In addition, we show that the knowledge boundary identified by our method for one VLLM can be used as a surrogate boundary for other VLLMs. Code will be released at https://github.com/Chord-Chen-30/VLLM-KnowledgeBoundary