Document-based Visual Question Answering poses a challenging task between linguistic sense disambiguation and fine-grained multimodal retrieval. Although there has been encouraging progress in document-based question answering due to the utilization of large language and open-world prior models\cite{1}, several challenges persist, including prolonged response times, extended inference durations, and imprecision in matching. In order to overcome these challenges, we propose Jaegar, a concatenation-based multi-transformer VQA model. To derive question features, we leverage the exceptional capabilities of RoBERTa large\cite{2} and GPT2-xl\cite{3} as feature extractors. Subsequently, we subject the outputs from both models to a concatenation process. This operation allows the model to consider information from diverse sources concurrently, strengthening its representational capability. By leveraging pre-trained models for feature extraction, our approach has the potential to amplify the performance of these models through concatenation. After concatenation, we apply dimensionality reduction to the output features, reducing the model's computational effectiveness and inference time. Empirical results demonstrate that our proposed model achieves competitive performance on Task C of the PDF-VQA Dataset. If the user adds any new data, they should make sure to style it as per the instructions provided in previous sections.
In Video Question Answering, videos are often processed as a full-length sequence of frames to ensure minimal loss of information. Recent works have demonstrated evidence that sparse video inputs are sufficient to maintain high performance. However, they usually discuss the case of single frame selection. In our work, we extend the setting to multiple number of inputs and other modalities. We characterize the task with different input sparsity and provide a tool for doing that. Specifically, we use a Gumbel-based learnable selection module to adaptively select the best inputs for the final task. In this way, we experiment over public VideoQA benchmarks and provide analysis on how sparsified inputs affect the performance. From our experiments, we have observed only 5.2%-5.8% loss of performance with only 10% of video lengths, which corresponds to 2-4 frames selected from each video. Meanwhile, we also observed the complimentary behaviour between visual and textual inputs, even under highly sparsified settings, suggesting the potential of improving data efficiency for video-and-language tasks.
정적 시각적 질의응답(VQA) 벤치마크는 빠르게 노후화된다. 항목이 학습 코퍼스에 누출되면 점수는 실제 시각 능력보다 암기를 반영할 수 있어 진정한 진보를 가린다. V*Bench와 같은 고품질 벤치마크를 재구축하려면 상당한 인간 주석이 필요하지만, 각 정적 릴리스는 곧 또 다른 누출 산출물이 될 수 있다. 우리는 평가 시점에 실제 이미지에서 답을 담고 있는 국소적 세부사항, 즉 시각 키(visual key)를 무작위로 재생성하는 라이브 벤치마크 프로토콜인 ReKey를 제안한다. 인간이 검증한 편집 슬롯(edit slot)을 사용하여 ReKey는 새로운 답, 구성에 기반한(construction-grounded) 레이블, 그리고 제어된 시각 탐색 난이도를 갖춘 새로운 인스턴스를 샘플링한다. V*Bench에서 ReKey가 재생성한 벤치마크는 8개의 최신 비전-언어 모델(VLM) 전반에 걸쳐 급격한 점수 상승을 드러낸다. 원본 항목은 재생성된 변형보다 9.5–18.8 percentage points 더 높은 점수를 기록한다. 시각 키를 재생 가능하게 함으로써 ReKey는 모델과 학습 데이터가 진화함에 따라 평가를 최신 상태로 유지한다.
Natural language rationales could provide intuitive, higher-level explanations that are easily understandable by humans, complementing the more broadly studied lower-level explanations based on gradients or attention weights. We present the first study focused on generating natural language rationales across several complex visual reasoning tasks: visual commonsense reasoning, visual-textual entailment, and visual question answering. The key challenge of accurate rationalization is comprehensive image understanding at all levels: not just their explicit content at the pixel level, but their contextual contents at the semantic and pragmatic levels. We present Rationale^VT Transformer, an integrated model that learns to generate free-text rationales by combining pretrained language models with object recognition, grounded visual semantic frames, and visual commonsense graphs. Our experiments show that the base pretrained language model benefits from visual adaptation and that free-text rationalization is a promising research direction to complement model interpretability for complex visual-textual reasoning tasks.
시각 질의응답(VQA)은 비전-언어 모델(VLM)에 상당한 계산 부담을 주며, 이로 인해 블랙박스 환경을 포함하여 다양한 VLM에 손쉽게 적용할 수 있는 모델 비종속적 입력 축소 방법이 요구된다. 시각 토큰 수에 단순히 상한을 두면 계산량은 줄지만 질문에 답하는 데 필요한 정보가 손실될 수 있다. 또한 단일한 축소 전략으로는 다양한 상황에 대응하기 어렵다. 우리의 핵심 관찰은 크로핑과 압축이 상보적이라는 것이다. 크로핑은 작은 대상과 국소적 세부 사항을 부각하는 반면, 전체 이미지 압축은 큰 대상이나 분산된 증거에 대한 문맥을 더 잘 보존한다. 이러한 관찰에 기반하여, 입력 이미지와 질문에 따라 크로핑 또는 압축을 적응적으로 선택하는 C$^2$-Routing을 제안한다. 두 전략 모두 핵심 정보를 잃을 수 있으므로, 원본 입력과 축소된 입력 각각에서의 정답 여부에 대한 예측 결합 분포를 이용해 폴백을 유도하고 정확도를 보호하는 안전 게이트를 추가로 설계한다. 소비자용 GPU의 자원 제약을 수용하기 위해 모든 학습 및 평가 실험은 RTX 3090 GPU에서 수행한다. 토큰 상한 적용 이상의 이득을 측정하기 위해, 시각 토큰 상한을 동일하게 1,024개로 둔 전체 이미지 기준선을 사용한다. 국소 세부 사항, 전역 장면, 장면 텍스트, 문서를 아우르는 3개의 VLM과 18개의 데이터셋에서 폭넓게 평가한다. 시각 토큰 예산이 이미 제한된 상태에서, C$^2$-Routing은 평균적으로 종단 간 FLOPs를 36.53%, 시각 토큰을 42.36% 추가로 줄이면서 과제 점수의 평균 상대 감소는 0.77%에 그친다.
Traditional machine learning models often require powerful hardware, making them unsuitable for deployment on resource-limited devices. Tiny Machine Learning (tinyML) has emerged as a promising approach for running machine learning models on these devices, but integrating multiple data modalities into tinyML models still remains a challenge due to increased complexity, latency, and power consumption. This paper proposes TinyVQA, a novel multimodal deep neural network for visual question answering tasks that can be deployed on resource-constrained tinyML hardware. TinyVQA leverages a supervised attention-based model to learn how to answer questions about images using both vision and language modalities. Distilled knowledge from the supervised attention-based VQA model trains the memory aware compact TinyVQA model and low bit-width quantization technique is employed to further compress the model for deployment on tinyML devices. The TinyVQA model was evaluated on the FloodNet dataset, which is used for post-disaster damage assessment. The compact model achieved an accuracy of 79.5%, demonstrating the effectiveness of TinyVQA for real-world applications. Additionally, the model was deployed on a Crazyflie 2.0 drone, equipped with an AI deck and GAP8 microprocessor. The TinyVQA model achieved low latencies of 56 ms and consumes 693 mW power while deployed on the tiny drone, showcasing its suitability for resource-constrained embedded systems.
Large-language models and large-vision models are increasingly capable of solving compositional reasoning tasks, as measured by breakthroughs in visual-question answering benchmarks. However, state-of-the-art solutions often involve careful construction of large pre-training and fine-tuning datasets, which can be expensive. The use of external tools, whether other ML models, search engines, or APIs, can significantly improve performance by breaking down high-level reasoning questions into sub-questions that are answerable by individual tools, but this approach has similar dataset construction costs to teach fine-tuned models how to use the available tools. We propose a technique in which existing training sets can be directly used for constructing computational environments with task metrics as rewards. This enables a model to autonomously teach itself to use itself or another model as a tool. By doing so, we augment training sets by integrating external signals. The proposed method starts with zero-shot prompts and iteratively refines them by selecting few-shot examples that maximize the task metric on the training set. Our experiments showcase how Gemini learns how to use itself, or another smaller and specialized model such as ScreenAI, to iteratively improve performance on training sets. Our approach successfully generalizes and improves upon zeroshot performance on charts, infographics, and document visual question-answering datasets
Vision-Language Models (VLMs) in visual question answering (VQA) offer a unique opportunity to enhance intra-operative decision-making, promote intuitive interactions, and significantly advancing surgical education. However, the development of VLMs for surgical VQA is challenging due to limited datasets and the risk of overfitting and catastrophic forgetting during full fine-tuning of pretrained weights. While parameter-efficient techniques like Low-Rank Adaptation (LoRA) and Matrix of Rank Adaptation (MoRA) address adaptation challenges, their uniform parameter distribution overlooks the feature hierarchy in deep networks, where earlier layers, that learn general features, require more parameters than later ones. This work introduces PitVQA++ with an open-ended PitVQA dataset and vector matrix-low-rank adaptation (Vector-MoLoRA), an innovative VLM fine-tuning approach for adapting GPT-2 to pituitary surgery. Open-Ended PitVQA comprises around 101,803 frames from 25 procedural videos with 745,972 question-answer sentence pairs, covering key surgical elements such as phase and step recognition, context understanding, tool detection, localization, and interactions recognition. Vector-MoLoRA incorporates the principles of LoRA and MoRA to develop a matrix-low-rank adaptation strategy that employs vector ranking to allocate more parameters to earlier layers, gradually reducing them in the later layers. Our approach, validated on the Open-Ended PitVQA and EndoVis18-VQA datasets, effectively mitigates catastrophic forgetting while significantly enhancing performance over recent baselines. Furthermore, our risk-coverage analysis highlights its enhanced reliability and trustworthiness in handling uncertain predictions. Our source code and dataset is available at~\url{https://github.com/HRL-Mike/PitVQA-Plus}.
텍스트, 표, 차트, 그림이 혼합된 수십에서 수백 페이지에 달하는 문서를 대상으로 하는 장문 시각 질의응답(VQA)은 일반적으로 '검색 후 독해(retrieve-then-read)' 파이프라인을 따릅니다. 본 연구의 설정에서는 병목 현상이 검색 재현율(recall)에서 리랭커 측의 근거 선택으로 이동합니다. MMLongBench-Doc에서 BGE-M3는 Recall@20 = 0.86에 도달하지만 F1@5는 0.254에 불과하며, 시각적 검색기인 ColPali조차 F1@5 = 0.332에 그칩니다. 원시 텍스트 스니펫만 확인하는 텍스트 전용 리랭크 대규모 언어 모델(LLM)은 상류 검색기가 이미지를 인코딩했더라도 표, 차트, 레이아웃 근거를 놓칩니다. 이에 따라 상호 보완적인 두 가지 구성 요소로 이루어진 Trident를 제안합니다. 첫째는 시각적 캡션, 섹션 경로, 엔티티 태그, 다축 개념 일치 항목, 텍스트 스니펫을 포함하여 각 후보를 LLM이 읽을 수 있는 의미론적 레코드로 변환한 후 단일 적응형 K(adaptive-K) 리랭크 호출을 수행하는 검색기 무관 LLM 리랭커인 Trident-R이며, 둘째는 종합 전에 주제, 엔티티, 구조적 관점에서 VLM에 프롬프트를 제공하는 생성 측 모듈인 Trident-S입니다. 두 가지 장문 문서 데이터셋에서 주석+리랭크 프로토콜은 5개의 이종 풀 전반에 걸쳐 검색 F1을 크게 향상시켰으며, 리랭크된 모든 풀은 가장 강력한 적응형 K 기준선(baseline)인 PageIndex를 넘어섰습니다. 주석이 없는 LLM 리랭크는 첫 번째 적중 순위를 거의 바꾸지 못하여 성능 향상이 구조화된 주석에서 비롯됨을 보여줍니다. Trident-S는 설계상 개방형 종합 질문을 목표로 하여 해당 질문에서 생성 정확도를 최대 6.6점 끌어올립니다. 최고의 Trident 구성은 본 평가에서 가장 강력한 다운스트림 QA 파이프라인이며, 두 LLM 심사위원 간의 순위 일치도(kappa = 0.913) 또한 일관되게 나타났습니다.
Although modern neural networks often generalize to new combinations of familiar concepts, the conditions that enable such compositionality have long been an open question. In this work, we study the systematicity gap in visual question answering: the performance difference between reasoning on previously seen and unseen combinations of object attributes. To test, we introduce a novel diagnostic dataset, CLEVR-HOPE. We find that the systematicity gap is not reduced by increasing the quantity of training data, but is reduced by increasing the diversity of training data. In particular, our experiments suggest that the more distinct attribute type combinations are seen during training, the more systematic we can expect the resulting model to be.
Large language models (LLMs) have achieved impressive pro-gress on several open-world tasks. Recently, using LLMs to build embodied agents has been a hotspot. This paper proposes STEVE, a comprehensive and visionary embodied agent in the Minecraft virtual environment. STEVE comprises three key components: vision perception, language instruction, and code action. Vision perception involves interpreting visual information in the environment, which is then integrated into the LLMs component with agent state and task instruction. Language instruction is responsible for iterative reasoning and decomposing complex tasks into manageable guidelines. Code action generates executable skill actions based on retrieval in skill database, enabling the agent to interact effectively within the Minecraft environment. We also collect STEVE-21K dataset, which includes 600+ vision-environment pairs, 20K knowledge question-answering pairs, and 200+ skill-code pairs. We conduct continuous block search, knowledge question and answering, and tech tree mastery to evaluate the performance. Extensive experiments show that STEVE achieves at most 1.5x faster unlocking key tech trees and 2.5x quicker in block search tasks.
Social intelligence, the ability to interpret emotions, intentions, and behaviors, is essential for effective communication and adaptive responses. As robots and AI systems become more prevalent in caregiving, healthcare, and education, the demand for AI that can interact naturally with humans grows. However, creating AI that seamlessly integrates multiple modalities, such as vision and speech, remains a challenge. Current video-based methods for social intelligence rely on general video recognition or emotion recognition techniques, often overlook the unique elements inherent in human interactions. To address this, we propose the Looped Video Debating (LVD) framework, which integrates Large Language Models (LLMs) with visual information, such as facial expressions and body movements, to enhance the transparency and reliability of question-answering tasks involving human interaction videos. Our results on the Social-IQ 2.0 benchmark show that LVD achieves state-of-the-art performance without fine-tuning. Furthermore, supplementary human annotations on existing datasets provide insights into the model's accuracy, guiding future improvements in AI-driven social intelligence.
볼륨(Volumetric) 의료 VQA는 길고 중복된 3D 시각 토큰(token) 시퀀스에 대한 추론을 요구하며, 특히 상호보완적인 모달리티가 다양한 진단 단서를 제공하지만 디코더를 반복되는 여러 해부학적 영역에 노출시키는 다중 시퀀스 MRI에서 더욱 그러합니다. 다중 시퀀스 시각적 중복 하에서의 추론을 연구하기 위해, 저자들은 먼저 12.9K명의 환자와 71.0K개의 시퀀스로부터 추출한 1.19M개의 QA 쌍으로 구성되어 자유 형식 텍스트와 객관식 질문을 모두 다루는 임상적으로 큐레이션된 유방 MRI 벤치마크인 BreMRIs-VQA를 소개합니다. 나아가 조밀한 볼륨을 모달리티별 프로토타입으로 압축하고 디코딩 중에 변화 인식 게이트 어텐션(attention)으로 상호보완적인 다수준 근거를 검색하는 선택적 시각 노출 프레임워크인 SeVeR를 제안하며, 이는 유용하지 않은 검색을 억제하는 한계 효용 자기일관성(marginal-utility self-consistency) 목적함수로 학습됩니다. BreMRIs-VQA 및 공개 벤치마크 실험을 통해 SeVeR가 훨씬 적은 시각 토큰만을 노출하면서도 판별 및 생성 성능을 모두 향상시킴을 입증합니다.
Technology and the fruition of cultural heritage are becoming increasingly more entwined, especially with the advent of smart audio guides, virtual and augmented reality, and interactive installations. Machine learning and computer vision are important components of this ongoing integration, enabling new interaction modalities between user and museum. Nonetheless, the most frequent way of interacting with paintings and statues still remains taking pictures. Yet images alone can only convey the aesthetics of the artwork, lacking is information which is often required to fully understand and appreciate it. Usually this additional knowledge comes both from the artwork itself (and therefore the image depicting it) and from an external source of knowledge, such as an information sheet. While the former can be inferred by computer vision algorithms, the latter needs more structured data to pair visual content with relevant information. Regardless of its source, this information still must be be effectively transmitted to the user. A popular emerging trend in computer vision is Visual Question Answering (VQA), in which users can interact with a neural network by posing questions in natural language and receiving answers about the visual content. We believe that this will be the evolution of smart audio guides for museum visits and simple image browsing on personal smartphones. This will turn the classic audio guide into a smart personal instructor with which the visitor can interact by asking for explanations focused on specific interests. The advantages are twofold: on the one hand the cognitive burden of the visitor will decrease, limiting the flow of information to what the user actually wants to hear; and on the other hand it proposes the most natural way of interacting with a guide, favoring engagement.
Purpose: To develop a bilingual multimodal visual question answering (VQA) benchmark for evaluating VLMs in ophthalmology. Methods: Ophthalmic image posts and associated captions published between January 1, 2016, and December 31, 2024, were collected from WeChat Official Accounts. Based on these captions, bilingual question-answer (QA) pairs in Chinese and English were generated using GPT-4o-mini. QA pairs were categorized into six subsets by question type and language: binary (Binary_CN, Binary_EN), single-choice (Single-choice_CN, Single-choice_EN), and open-ended (Open-ended_CN, Open-ended_EN). The benchmark was used to evaluate the performance of three VLMs: GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL-72B-Instruct. Results: The final OphthalWeChat dataset included 3,469 images and 30,120 QA pairs across 9 ophthalmic subspecialties, 548 conditions, 29 imaging modalities, and 68 modality combinations. Gemini 2.0 Flash achieved the highest overall accuracy (0.548), outperforming GPT-4o (0.522, P < 0.001) and Qwen2.5-VL-72B-Instruct (0.514, P < 0.001). It also led in both Chinese (0.546) and English subsets (0.550). Subset-specific performance showed Gemini 2.0 Flash excelled in Binary_CN (0.687), Single-choice_CN (0.666), and Single-choice_EN (0.646), while GPT-4o ranked highest in Binary_EN (0.717), Open-ended_CN (BLEU-1: 0.301; BERTScore: 0.382), and Open-ended_EN (BLEU-1: 0.183; BERTScore: 0.240). Conclusions: This study presents the first bilingual VQA benchmark for ophthalmology, distinguished by its real-world context and inclusion of multiple examinations per patient. The dataset reflects authentic clinical decision-making scenarios and enables quantitative evaluation of VLMs, supporting the development of accurate, specialized, and trustworthy AI systems for eye care.
Recent advances in multimodal question answering have primarily focused on combining heterogeneous modalities or fine-tuning multimodal large language models. While these approaches have shown strong performance, they often rely on a single, generalized reasoning strategy, overlooking the unique characteristics of each modality ultimately limiting both accuracy and interpretability. To address these limitations, we propose MAMMQA, a multi-agent QA framework for multimodal inputs spanning text, tables, and images. Our system includes two Visual Language Model (VLM) agents and one text-based Large Language Model (LLM) agent. The first VLM decomposes the user query into sub-questions and sequentially retrieves partial answers from each modality. The second VLM synthesizes and refines these results through cross-modal reasoning. Finally, the LLM integrates the insights into a cohesive answer. This modular design enhances interpretability by making the reasoning process transparent and allows each agent to operate within its domain of expertise. Experiments on diverse multimodal QA benchmarks demonstrate that our cooperative, multi-agent framework consistently outperforms existing baselines in both accuracy and robustness.
Current Artificial Intelligence (AI) methods, most based on deep learning, have facilitated progress in several fields, including computer vision and natural language understanding. The progress of these AI methods is measured using benchmarks designed to solve challenging tasks, such as visual question answering. A question remains of how much understanding is leveraged by these methods and how appropriate are the current benchmarks to measure understanding capabilities. To answer these questions, we have analysed existing benchmarks and their understanding capabilities, defined by a set of understanding capabilities, and current research streams. We show how progress has been made in benchmark development to measure understanding capabilities of AI methods and we review as well how current methods develop understanding capabilities.
We introduce a vision-language foundation model called VL-BEiT, which is a bidirectional multimodal Transformer learned by generative pretraining. Our minimalist solution conducts masked prediction on both monomodal and multimodal data with a shared Transformer. Specifically, we perform masked vision-language modeling on image-text pairs, masked language modeling on texts, and masked image modeling on images. VL-BEiT is learned from scratch with one unified pretraining task, one shared backbone, and one-stage training. Our method is conceptually simple and empirically effective. Experimental results show that VL-BEiT obtains strong results on various vision-language benchmarks, such as visual question answering, visual reasoning, and image-text retrieval. Moreover, our method learns transferable visual features, achieving competitive performance on image classification, and semantic segmentation.
In recent years, vision-language research has shifted to study tasks which require more complex reasoning, such as interactive question answering, visual common sense reasoning, and question-answer plausibility prediction. However, the datasets used for these problems fail to capture the complexity of real inputs and multimodal environments, such as ambiguous natural language requests and diverse digital domains. We introduce Mobile app Tasks with Iterative Feedback (MoTIF), a dataset with natural language commands for the greatest number of interactive environments to date. MoTIF is the first to contain natural language requests for interactive environments that are not satisfiable, and we obtain follow-up questions on this subset to enable research on task uncertainty resolution. We perform initial feasibility classification experiments and only reach an F1 score of 37.3, verifying the need for richer vision-language representations and improved architectures to reason about task feasibility.
Out-of-distribution (OOD) testing is increasingly popular for evaluating a machine learning system's ability to generalize beyond the biases of a training set. OOD benchmarks are designed to present a different joint distribution of data and labels between training and test time. VQA-CP has become the standard OOD benchmark for visual question answering, but we discovered three troubling practices in its current use. First, most published methods rely on explicit knowledge of the construction of the OOD splits. They often rely on ``inverting'' the distribution of labels, e.g. answering mostly 'yes' when the common training answer is 'no'. Second, the OOD test set is used for model selection. Third, a model's in-domain performance is assessed after retraining it on in-domain splits (VQA v2) that exhibit a more balanced distribution of labels. These three practices defeat the objective of evaluating generalization, and put into question the value of methods specifically designed for this dataset. We show that embarrassingly-simple methods, including one that generates answers at random, surpass the state of the art on some question types. We provide short- and long-term solutions to avoid these pitfalls and realize the benefits of OOD evaluation.