Zero-shot learning (ZSL) is a popular research problem that aims at predicting for those classes that have never appeared in the training stage by utilizing the inter-class relationship with some side information. In this study, we propose to model the compositional and expressive semantics of class labels by an OWL (Web Ontology Language) ontology, and further develop a new ZSL framework with ontology embedding. The effectiveness has been verified by some primary experiments on animal image classification and visual question answering.
We present in this paper a novel scheme for multimodal learning named the Parallel Attention mechanism. In addition, to take into account the advantages of grammar and context in Vietnamese, we propose the Hierarchical Linguistic Features Extractor instead of using an LSTM network to extract linguistic features. Based on these two novel modules, we introduce the Parallel Attention Transformer (PAT), achieving the best accuracy compared to all baselines on the benchmark ViVQA dataset and other SOTA methods including SAAA and MCAN.
Teaching Visual Question Answering (VQA) models to refrain from answering unanswerable questions is necessary for building a trustworthy AI system. Existing studies, though have explored various aspects of VQA but somewhat ignored this particular attribute. This paper aims to bridge the research gap by contributing a comprehensive dataset, called UNK-VQA. The dataset is specifically designed to address the challenge of questions that models do not know. To this end, we first augment the existing data via deliberate perturbations on either the image or question. In specific, we carefully ensure that the question-image semantics remain close to the original unperturbed distribution. By this means, the identification of unanswerable questions becomes challenging, setting our dataset apart from others that involve mere image replacement. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models and discover their significant limitations when applied to our dataset. Additionally, we also propose a straightforward method to tackle these unanswerable questions. This dataset, we believe, will serve as a valuable benchmark for enhancing the abstention capability of VQA models, thereby leading to increased trustworthiness of AI systems. We have made the dataset (https://github.com/guoyang9/UNK-VQA) available to facilitate further exploration in this area.
Agentic video reasoning systems gather and assess video evidence across stages, selecting relevant spans as visual evidence for answering questions. New candidates and observations prompt reassessment of the current span, but revisions can correct localization errors or discard useful evidence already present in the adopted span. Systems therefore need to connect evidence organization across stages with temporal state control, allowing accumulated information to jointly inform revision assessment while explicit adoption decisions govern how assessment outcomes affect downstream visual evidence. We introduce VideoSlate, a stateful commit-control framework built around shared evidence. A typed evidence blackboard links candidate spans, provenance, construction, and verifier observations, enabling an incumbent-relative certifier to assess revision support from the candidate, current span, and related proposals. Based on certification, the controller authorizes a revision or retains the current span, separating evidence accumulation from state updates and recording each decision and its evidence on the blackboard. Extensive evaluations across video grounding and grounded question-answering benchmarks show that VideoSlate can improve temporal localization through selective revision.
Audio-Visual Question Answering (AVQA) is a complex multi-modal reasoning task, demanding intelligent systems to accurately respond to natural language queries based on audio-video input pairs. Nevertheless, prevalent AVQA approaches are prone to overlearning dataset biases, resulting in poor robustness. Furthermore, current datasets may not provide a precise diagnostic for these methods. To tackle these challenges, firstly, we propose a novel dataset, MUSIC-AVQA-R, crafted in two steps: rephrasing questions within the test split of a public dataset (MUSIC-AVQA) and subsequently introducing distribution shifts to split questions. The former leads to a large, diverse test space, while the latter results in a comprehensive robustness evaluation on rare, frequent, and overall questions. Secondly, we propose a robust architecture that utilizes a multifaceted cycle collaborative debiasing strategy to overcome bias learning. Experimental results show that this architecture achieves state-of-the-art performance on MUSIC-AVQA-R, notably obtaining a significant improvement of 9.32%. Extensive ablation experiments are conducted on the two datasets mentioned to analyze the component effectiveness within the debiasing strategy. Additionally, we highlight the limited robustness of existing multi-modal QA methods through the evaluation on our dataset. We also conduct experiments combining various baselines with our proposed strategy on two datasets to verify its plug-and-play capability. Our dataset and code are available at https://github.com/reml-group/MUSIC-AVQA-R.
An important goal of computer vision is to build systems that learn visual representations over time that can be applied to many tasks. In this paper, we investigate a vision-language embedding as a core representation and show that it leads to better cross-task transfer than standard multi-task learning. In particular, the task of visual recognition is aligned to the task of visual question answering by forcing each to use the same word-region embeddings. We show this leads to greater inductive transfer from recognition to VQA than standard multitask learning. Visual recognition also improves, especially for categories that have relatively few recognition training labels but appear often in the VQA setting. Thus, our paper takes a small step towards creating more general vision systems by showing the benefit of interpretable, flexible, and trainable core representations.
Large Language Models (LLMs) demonstrate impressive reasoning ability and the maintenance of world knowledge not only in natural language tasks, but also in some vision-language tasks such as open-domain knowledge-based visual question answering (OK-VQA). As images are invisible to LLMs, researchers convert images to text to engage LLMs into the visual question reasoning procedure. This leads to discrepancies between images and their textual representations presented to LLMs, which consequently impedes final reasoning performance. To fill the information gap and better leverage the reasoning capability, we design a framework that enables LLMs to proactively ask relevant questions to unveil more details in the image, along with filters for refining the generated information. We validate our idea on OK-VQA and A-OKVQA. Our method continuously boosts the performance of baselines methods by an average gain of 2.15% on OK-VQA, and achieves consistent improvements across different LLMs.
Document Question Answering (QA) presents a challenge in understanding visually-rich documents (VRD), particularly those dominated by lengthy textual content like research journal articles. Existing studies primarily focus on real-world documents with sparse text, while challenges persist in comprehending the hierarchical semantic relations among multiple pages to locate multimodal components. To address this gap, we propose PDF-MVQA, which is tailored for research journal articles, encompassing multiple pages and multimodal information retrieval. Unlike traditional machine reading comprehension (MRC) tasks, our approach aims to retrieve entire paragraphs containing answers or visually rich document entities like tables and figures. Our contributions include the introduction of a comprehensive PDF Document VQA dataset, allowing the examination of semantically hierarchical layout structures in text-dominant documents. We also present new VRD-QA frameworks designed to grasp textual contents and relations among document layouts simultaneously, extending page-level understanding to the entire multi-page document. Through this work, we aim to enhance the capabilities of existing vision-and-language models in handling challenges posed by text-dominant documents in VRD-QA.
Marine video understanding is crucial for a wide range of real-world applications. However, existing marine video datasets suffer from limitations, including the limited number of data samples, fine-grained text descriptions, and detailed visual attributes. To address these limitations, we introduce a large-scale, video-text dataset featuring rich, in-the-wild visual attributes tailored for complex marine environments. Our dataset comprises 2,000 video-text pairs annotated with 13 attributes covering up to four objects per scene, object visual/geometric properties, behaviors, environmental context, and camera motion, spanning diverse lighting conditions, times of day, and 7 countries. From these annotations, we generate 16,080 video-question pairs (12,080 multiple-choice, 4,000 open-ended). Furthermore, we benchmark both proprietary and open-source video-LLM models on video question answering and video captioning tasks. We find that (i) proprietary models often achieves superior performances; (ii) VLMs handle object motion well but struggle with complex camera motion; (iii) closed-source models struggle to perceive camera motion; and (iv) VLMs struggle to reason in static cameras.
Traditional Chinese Medicine (TCM), as an effective alternative medicine, has been receiving increasing attention. In recent years, the rapid development of large language models (LLMs) tailored for TCM has underscored the need for an objective and comprehensive evaluation framework to assess their performance on real-world tasks. However, existing evaluation datasets are limited in scope and primarily text-based, lacking a unified and standardized multimodal question-answering (QA) benchmark. To address this issue, we introduce TCM-Ladder, the first multimodal QA dataset specifically designed for evaluating large TCM language models. The dataset spans multiple core disciplines of TCM, including fundamental theory, diagnostics, herbal formulas, internal medicine, surgery, pharmacognosy, and pediatrics. In addition to textual content, TCM-Ladder incorporates various modalities such as images and videos. The datasets were constructed using a combination of automated and manual filtering processes and comprise 52,000+ questions in total. These questions include single-choice, multiple-choice, fill-in-the-blank, diagnostic dialogue, and visual comprehension tasks. We trained a reasoning model on TCM-Ladder and conducted comparative experiments against 9 state-of-the-art general domain and 5 leading TCM-specific LLMs to evaluate their performance on the datasets. Moreover, we propose Ladder-Score, an evaluation method specifically designed for TCM question answering that effectively assesses answer quality regarding terminology usage and semantic expression. To our knowledge, this is the first work to evaluate mainstream general domain and TCM-specific LLMs on a unified multimodal benchmark. The datasets and leaderboard are publicly available at https://tcmladder.com or https://54.211.107.106 and will be continuously updated.
Autonomous aerial monitoring is an important task aimed at gathering information from areas that may not be easily accessible by humans. At the same time, this task often requires recognizing anomalies from a significant distance or not previously encountered in the past. In this paper, we propose a novel framework that leverages the advanced capabilities provided by Large Language Models (LLMs) to actively collect information and perform anomaly detection in novel scenes. To this end, we propose an LLM based model dialogue approach, in which two deep learning models engage in a dialogue to actively control a drone to increase perception and anomaly detection accuracy. We conduct our experiments in a high fidelity simulation environment where an LLM is provided with a predetermined set of natural language movement commands mapped into executable code functions. Additionally, we deploy a multimodal Visual Question Answering (VQA) model charged with the task of visual question answering and captioning. By engaging the two models in conversation, the LLM asks exploratory questions while simultaneously flying a drone into different parts of the scene, providing a novel way to implement active perception. By leveraging LLMs reasoning ability, we output an improved detailed description of the scene going beyond existing static perception approaches. In addition to information gathering, our approach is utilized for anomaly detection and our results demonstrate the proposed methods effectiveness in informing and alerting about potential hazards.
각 답변을 뒷받침하는 정확한 문서 영역을 찾는 명시적 시각 그라운딩(grounding)을 갖춘 효율적인 멀티모달 문서 질의응답은 여전히 해결되지 않은 과제이다. 현재 접근법은 대규모 주석 데이터셋이 필요하고 최적화 정체에 이르는 지도 미세조정(SFT), 그리고 명확한 이점 없이 장황한 중간 추적(trace)에 의존하여 추론 토큰 비용을 부풀리는 추론 중심 강화학습(RL)으로 나뉜다. 우리는 Perception-RFT를 제안한다. 이는 멀티모달 문서 QA에 Group Relative Policy Optimization(GRPO)을 적용하는 학습 프레임워크로, 중간 추론 토큰을 우회하여 시각 특징을 구조화된 그라운딩 출력과 직접 정렬한다. 추론의 필요성을 엄밀히 평가하기 위해 동일한 보상 설정 하에서 추론 변형을 구성한다. 추론이 활성화된 모델은 학습 중 추론 추적을 억제하며 4B 매개변수 규모에서 직접 지각(perception) 기반 정책으로 수렴하고, 쿼리당 추론 토큰 길이를 60% 이상 줄이는 반면, 추론 활성화 RL은 지각 전용 학습보다 성능이 낮다. Qwen3-VL-4B 최적화 동역학에 대한 세밀한 분석을 통해, 텍스트 도메인 사후 학습에서 확인된 SFT 포화와 cold-start RL 불안정성이 멀티모달로 확장됨을 확인하고, 공동 RL 최적화 하에서 두 분포 외(OOD) 벤치마크(4,828 샘플)에서 의미론적 견고성과 기하학적 정밀도 사이의 선택적 trade-off인 Grounding Divergence를 새롭게 규명한다. 또한 초기 SFT→RL 전환이 65% 적은 학습 데이터로 동등한 정밀도를 달성함을 보인다.
비전-언어 모델(VLMs)은 차트 질의응답에 강력하지만, 많은 질문이 광학 문자 인식(OCR) 텍스트와 경량 언어 추론만으로 답변 가능할 때 모든 쿼리에 대해 VLM을 호출하는 것은 불필요하게 비쌀 수 있습니다. 우리는 비용 적응형 차트 질의응답을 위한 대화형 시스템인 SAFE-Cascade를 제시합니다. 차트 이미지와 자연어 질문이 주어지면, SAFE-Cascade는 먼저 OCR을 사용하여 차트 텍스트를 추출하고 텍스트 전용 언어 모델로부터 잠정적 답변을 얻은 다음, 학습된 라우터를 사용하여 텍스트 답변을 수락할지 VLM으로 이관할지 결정합니다. 이 데모는 사용자에게 이러한 의사결정 과정을 보여줍니다: OCR 증거, 텍스트 전용 답변, 라우팅 확률, 이관 결정, 최종 답변, 추정 비용, 추정 지연 시간이 나란히 표시됩니다. SAFE-Cascade는 시각적 그라운딩(grounding)이 실제로 언제 필요한지 이해하기 위한 투명한 인터페이스로 설계되었습니다. 사용자는 차트를 업로드하거나 선택하고, 질문하고, 각 경로가 사용한 증거를 검사하며, 텍스트 전용 답변과 VLM 답변을 비교하고, 정확도-비용 프론티어를 탐색하기 위해 이관 임계값을 조정할 수 있습니다. 이 시스템은 OCR에 Azure Document Intelligence를, 텍스트 전용 모델로 gpt-5-mini를, VLM으로 gemini-2.5-flash-image를 사용하며, 추론 시간 특징으로 학습된 랜덤 포레스트 라우터를 사용하여 구현되었습니다. 2,500개 예제 실험 중 375개 예제로 구성된 별도의 ChartQA 테스트 분할에서, SAFE-Cascade는 전체 VLM 기준선(baseline)의 67.7% 정확도 및 100% VLM 호출률과 비교하여 69.1%의 통합 정확도와 73.1%의 VLM 호출률을 달성했습니다. 관찰된 +1.4%포인트의 차이는 통계적으로 불확실하므로, 우리는 SAFE-Cascade가 VLM 호출을 26.9%, 추정 비용을 9.3% 줄이면서 전체 VLM 성능과 대등한 것으로 해석합니다. 이 데모는 선택적 모달리티 라우팅이 멀티모달 지식 시스템을 더 투명하고, 조정 가능하며, 비용 인식형(cost-aware)으로 만들 수 있는 방법을 보여줍니다.
Prior benchmarks have analyzed models' answers to questions about videos in order to measure visual compositional reasoning. Action Genome Question Answering (AGQA) is one such benchmark. AGQA provides a training/test split with balanced answer distributions to reduce the effect of linguistic biases. However, some biases remain in several AGQA categories. We introduce AGQA 2.0, a version of this benchmark with several improvements, most namely a stricter balancing procedure. We then report results on the updated benchmark for all experiments.
장편 비디오를 이해하는 데 가장 어려운 부분은 어느 한 순간을 정확히 짚어내는 것이 아니라 여러 순간을 종합하는 것이다. 예를 들어 수 시간 분량의 콘텐츠에 걸쳐 전개되는 내용을 추적하고 각 조각이 어떻게 맞물리는지 설명해야 하는 질문에 답하는 일이 그렇다. 지금까지 대부분의 장편 비디오 QA 데이터셋은 짧은 답변이나 객관식 선택만 요구하여 장편 비디오 이해 과제를 검색 수준으로 지나치게 단순화한다. 우리는 마음 이론 질문을 기반으로 장편 서사 비디오에 대한 장문 개방형 질의응답을 위한 데이터셋 LF-VideoQA를 소개한다. 등장인물의 믿음은 그가 보았거나 전해 들은 모든 것에 좌우되므로, 각 질문은 비디오 전반에 분포한 증거를 모아 추론할 것을 요구한다. 이 데이터셋은 최근 영화 94편에 대한 6,300개 이상의 질문을 포함하며, 각 질문은 평균 23분에 걸친 장면에 대한 추론을 요구하고 재현율과 정밀도 채점을 위한 질문별 루브릭이 함께 제공된다. 데이터셋은 시나리오에 장면 사건과 각 등장인물의 1차 믿음 및 메타 믿음을 주석한 뒤, LLM 기반 질문 생성과 검증을 통해 (질문, 답변, 루브릭) 삼중쌍을 생성하는 방식으로 완전 자동으로 만들어지며, 사람 대상 연구로 그 품질을 검증한다. 우리는 세 가지 설정에서 네 계열의 최첨단 오픈소스 VLM을 평가했으며 이 과제가 매우 어렵다는 것을 확인했다. (i) 시나리오에 접근할 수 있는 오라클조차 완전한 답변에는 크게 못 미친다. (ii) 오라클이 아닌 최고 모델은 오라클 재현율 수준의 절반을 간신히 달성한다. (iii) 기존 장편 비디오 벤치마크와 달리 캡션과 자막은 비디오 자체보다 성능이 낮으며, 이는 질문이 시각적 증거를 필요로 함을 나타낸다. (iv) 재현율이 지배적인 병목이며, 답변이 요구되는 루브릭 주장을 다수 누락한다.
Many vision and language models suffer from poor visual grounding - often falling back on easy-to-learn language priors rather than basing their decisions on visual concepts in the image. In this work, we propose a generic approach called Human Importance-aware Network Tuning (HINT) that effectively leverages human demonstrations to improve visual grounding. HINT encourages deep networks to be sensitive to the same input regions as humans. Our approach optimizes the alignment between human attention maps and gradient-based network importances - ensuring that models learn not just to look at but rather rely on visual concepts that humans found relevant for a task when making predictions. We apply HINT to Visual Question Answering and Image Captioning tasks, outperforming top approaches on splits that penalize over-reliance on language priors (VQA-CP and robust captioning) using human attention demonstrations for just 6% of the training data.
The intersection of vision and language is of major interest due to the increased focus on seamless integration between recognition and reasoning. Scene graphs (SGs) have emerged as a useful tool for multimodal image analysis, showing impressive performance in tasks such as Visual Question Answering (VQA). In this work, we demonstrate that despite the effectiveness of scene graphs in VQA tasks, current methods that utilize idealized annotated scene graphs struggle to generalize when using predicted scene graphs extracted from images. To address this issue, we introduce the SelfGraphVQA framework. Our approach extracts a scene graph from an input image using a pre-trained scene graph generator and employs semantically-preserving augmentation with self-supervised techniques. This method improves the utilization of graph representations in VQA tasks by circumventing the need for costly and potentially biased annotated data. By creating alternative views of the extracted graphs through image augmentations, we can learn joint embeddings by optimizing the informational content in their representations using an un-normalized contrastive approach. As we work with SGs, we experiment with three distinct maximization strategies: node-wise, graph-wise, and permutation-equivariant regularization. We empirically showcase the effectiveness of the extracted scene graph for VQA and demonstrate that these approaches enhance overall performance by highlighting the significance of visual information. This offers a more practical solution for VQA tasks that rely on SGs for complex reasoning questions.
Visual Dialog is a challenging vision-language task since the visual dialog agent needs to answer a series of questions after reasoning over both the image content and dialog history. Though existing methods try to deal with the cross-modal understanding in visual dialog, they are still not enough in ranking candidate answers based on their understanding of visual and textual contexts. In this paper, we analyze the cross-modal understanding in visual dialog based on the vision-language pre-training model VD-BERT and propose a novel approach to improve the cross-modal understanding for visual dialog, named ICMU. ICMU enhances cross-modal understanding by distinguishing different pulled inputs (i.e. pulled images, questions or answers) based on four-way contrastive learning. In addition, ICMU exploits the single-turn visual question answering to enhance the visual dialog model's cross-modal understanding to handle a multi-turn visually-grounded conversation. Experiments show that the proposed approach improves the visual dialog model's cross-modal understanding and brings satisfactory gain to the VisDial dataset.
Rebus puzzles, visual riddles that encode language through imagery, spatial arrangement, and symbolic substitution, pose a unique challenge to current vision-language models (VLMs). Unlike traditional image captioning or question answering tasks, rebus solving requires multi-modal abstraction, symbolic reasoning, and a grasp of cultural, phonetic and linguistic puns. In this paper, we investigate the capacity of contemporary VLMs to interpret and solve rebus puzzles by constructing a hand-generated and annotated benchmark of diverse English-language rebus puzzles, ranging from simple pictographic substitutions to spatially-dependent cues ("head" over "heels"). We analyze how different VLMs perform, and our findings reveal that while VLMs exhibit some surprising capabilities in decoding simple visual clues, they struggle significantly with tasks requiring abstract reasoning, lateral thinking, and understanding visual metaphors.
HumaniBench는 대규모 다중모달 모델(LMM)의 인간 중심 평가를 위한 종합 벤치마크로, 공정성, 윤리성, 공감능력, 포용성 등 7가지 인간 중심 AI(HCAI) 원칙을 포함한 32,000개의 실제 이미지-질문 쌍으로 구성되어 있습니다. GPT4o 기반의 확장 가능한 주석 파이프라인과 전문가 검증을 통해 데이터가 구축되었으며, 시각적 질문응답(VQA), 다국어 QA, 시각적 근거 찾기, 공감 캡셔닝, 강인성 테스트 등 7가지 다양한 과제를 포함합니다. 15개의 최신 LMM을 평가한 결과, 상용 모델이 전반적으로 우수하지만 강인성과 시각적 근거 찾기에서 약점을 보였고, 일부 오픈소스 모델은 정확성과 인간 중심 원칙 준수 간 균형에 어려움을 겪는 것으로 나타났습니다. HumaniBench는 HCAI 원칙에 특화된 최초의 벤치마크로, 모델의 사회적 책임과 정확성 간 정렬 문제를 진단하고 개선 방향을 제시하는 데 기여합니다.