The goal of combining the robustness of neural networks and the expressiveness of symbolic methods has rekindled the interest in Neuro-Symbolic AI. Deep Probabilistic Programming Languages (DPPLs) have been developed for probabilistic logic programming to be carried out via the probability estimations of deep neural networks. However, recent SOTA DPPL approaches allow only for limited conditional probabilistic queries and do not offer the power of true joint probability estimation. In our work, we propose an easy integration of tractable probabilistic inference within a DPPL. To this end, we introduce SLASH, a novel DPPL that consists of Neural-Probabilistic Predicates (NPPs) and a logic program, united via answer set programming (ASP). NPPs are a novel design principle allowing for combining all deep model types and combinations thereof to be represented as a single probabilistic predicate. In this context, we introduce a novel $+/-$ notation for answering various types of probabilistic queries by adjusting the atom notations of a predicate. To scale well, we show how to prune the stochastically insignificant parts of the (ground) program, speeding up reasoning without sacrificing the predictive performance. We evaluate SLASH on a variety of different tasks, including the benchmark task of MNIST addition and Visual Question Answering (VQA).
스트리밍 비디오 이해(SVU)는 모델이 지속적으로 전개되는 비디오 스트림에 대한 사용자 질의에 답할 것을 요구한다. 오프라인 비디오 QA와 달리 SVU는 근거가 과거에 있거나, 현재 장면에 나타나거나, 미래의 관찰 이후에야 나타나는 질의를 처리해야 한다. 기존 스트리밍 비디오 시스템은 흔히 압축된 시각 메모리나 고정된 근거 접근 방식에 의존하는데, 이는 세밀한 세부 정보를 잃고 각 질의의 시간적 의도에 맞게 답변 과정을 적응시키지 못할 수 있다. 본 논문에서는 시간 사전(Time Dictionary)을 이용해 시간 전략 계획을 수행하는 스트리밍 VQA 프레임워크인 TimeTraveler를 제안한다. 스트림의 서로 다른 시간 영역을 탐색해야 한다는 필요성에서 착안하여, TimeTraveler는 관찰된 각 순간을 타임스탬프로 색인된 구조화 캡션으로 저장하고 이 사전을 질의 가능한 명시적 근거 출처로 사용한다. 질의가 주어지면 TimeTraveler는 과거 근거를 상기할지, 현재 맥락을 읽을지, 미래 관찰을 기다릴지를 결정한 다음 전략별 근거 획득 과정을 적용한다. 스트리밍 비디오 QA 벤치마크에 대한 비교 실험은 TimeTraveler가 세밀한 시간적 근거를 보존하고 각 질의에 적절한 시간 전략을 선택함으로써 스트리밍 질의응답을 향상시킴을 보여준다. 프로젝트 페이지는 https://geppa.github.io/TimeTraveler 에서 확인할 수 있다.
Zero-shot anomaly detection (ZSAD) identifies anomalies without needing training samples from the target dataset, essential for scenarios with privacy concerns or limited data. Vision-language models like CLIP show potential in ZSAD but have limitations: relying on manually crafted fixed textual descriptions or anomaly prompts is time-consuming and prone to semantic ambiguity, and CLIP struggles with pixel-level anomaly segmentation, focusing more on global semantics than local details. To address these limitations, We introduce KAnoCLIP, a novel ZSAD framework that leverages vision-language models. KAnoCLIP combines general knowledge from a Large Language Model (GPT-3.5) and fine-grained, image-specific knowledge from a Visual Question Answering system (Llama3) via Knowledge-Driven Prompt Learning (KnPL). KnPL uses a knowledge-driven (KD) loss function to create learnable anomaly prompts, removing the need for fixed text prompts and enhancing generalization. KAnoCLIP includes the CLIP visual encoder with V-V attention (CLIP-VV), Bi-Directional Cross-Attention for Multi-Level Cross-Modal Interaction (Bi-CMCI), and Conv-Adapter. These components preserve local visual semantics, improve local cross-modal fusion, and align global visual features with textual information, enhancing pixel-level anomaly detection. KAnoCLIP achieves state-of-the-art performance in ZSAD across 12 industrial and medical datasets, demonstrating superior generalization compared to existing methods.
기존 자기 진화 LMM은 답변 일치만 최적화해 디코더가 시각 토큰보다 언어 사전분포에 의존하는 시각적 언더컨디셔닝 문제가 있다. VISE는 기하·의미 불변성 보상으로 시각 조건화를 직접 정규화하는 비지도 자기 진화 프레임워크로, Qwen3-VL-2B 기준 COCO·TextCaps CIDEr가 각각 +16.85·+19.66 향상되고 Chair-I 객체 환각이 5.0포인트 감소했다.
Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs). Although promising, existing heuristic mRAGs typically predefined fixed retrieval processes, which causes two issues: (1) Non-adaptive Retrieval Queries. (2) Overloaded Retrieval Queries. However, these flaws cannot be adequately reflected by current knowledge-seeking visual question answering (VQA) datasets, since the most required knowledge can be readily obtained with a standard two-step retrieval. To bridge the dataset gap, we first construct Dyn-VQA dataset, consisting of three types of "dynamic" questions, which require complex knowledge retrieval strategies variable in query, tool, and time: (1) Questions with rapidly changing answers. (2) Questions requiring multi-modal knowledge. (3) Multi-hop questions. Experiments on Dyn-VQA reveal that existing heuristic mRAGs struggle to provide sufficient and precisely relevant knowledge for dynamic questions due to their rigid retrieval processes. Hence, we further propose the first self-adaptive planning agent for multimodal retrieval, OmniSearch. The underlying idea is to emulate the human behavior in question solution which dynamically decomposes complex multimodal questions into sub-question chains with retrieval action. Extensive experiments prove the effectiveness of our OmniSearch, also provide direction for advancing mRAG. The code and dataset will be open-sourced at https://github.com/Alibaba-NLP/OmniSearch.
대규모 언어 모델(LLM)과 비전-언어 모델(VLM)의 최근 발전은 체화된 AI(embodied AI) 및 로봇 인식의 핵심 역량인 3차원 질의응답(3D-QA)에 새로운 가능성을 열었습니다. 그러나 기존 방식 대부분은 비용이 많이 드는 주석을 활용한 3D 전용 학습(training)이나 미세조정(fine-tuning)에 의존하여 확장성과 실제 환경 적용성이 제한됩니다. 본 논문에서는 완전한 3D 재구성 없이도 장면에 대한 다중 뷰 관측을 통해 3차원 공간 추론을 수행하는, 완전한 비학습 기반(training-free) 모듈형 프레임워크인 ViewMind3D를 제안합니다. 이 프레임워크는 3D-QA 태스크를 네 가지 해석 가능한 구성 요소로 분해합니다: (1) 질문 기반 다중 뷰 선택, (2) 언어 조건부 객체 단서를 활용한 유도된 시각적 그라운딩(visual grounding), (3) 조감도(BEV) 시점 지표를 통한 공간적 문맥 인코딩, (4) 역할 기반 추론(inference)을 통한 구조화된 답변 생성. 이러한 설계를 통해 모델 튜닝 없이도 구조화되고 강건하며 해석 가능한 추론이 가능해집니다. ScanQA 및 SQA3D에 대한 실험 결과, ViewMind3D는 기존 비학습 기반 모델 및 미세조정된 3D-LLM과 비교해 경쟁력 있는 성능을 달성함을 보여줍니다. 특히 본 제안 방법은 높은 전체 정확도(50.8%)를 유지하고 ScanQA에서 73.4 CIDEr를 달성하면서, SQA3D의 'What' 질문과 같이 공간적으로 그라운딩된 질문 유형에서 성능을 향상시킵니다. 이러한 결과는 범용 LLM과 VLM의 모듈형 오케스트레이션을 통해 실제 환경에서의 로봇 인식을 위한 효과적인 3D 추론을 달성할 수 있음을 입증합니다.
Music performances are representative scenarios for audio-visual modeling. Unlike common scenarios with sparse audio, music performances continuously involve dense audio signals throughout. While existing multimodal learning methods on the audio-video QA demonstrate impressive capabilities in general scenarios, they are incapable of dealing with fundamental problems within the music performances: they underexplore the interaction between the multimodal signals in performance and fail to consider the distinctive characteristics of instruments and music. Therefore, existing methods tend to answer questions regarding musical performances inaccurately. To bridge the above research gaps, (i) given the intricate multimodal interconnectivity inherent to music data, our primary backbone is designed to incorporate multimodal interactions within the context of music; (ii) to enable the model to learn music characteristics, we annotate and release rhythmic and music sources in the current music datasets; (iii) for time-aware audio-visual modeling, we align the model's music predictions with the temporal dimension. Our experiments show state-of-the-art effects on the Music AVQA datasets. Our code is available at https://github.com/xid32/Amuse.
최근 비전-언어 모델(VLM)은 고품질 이미지-텍스트 쌍의 가용성 증가에 힘입어 시각적 이해에서 뛰어난 성과를 거두었다. 그러나 VLM의 성능은 시간적 이해가 필수적이고 대규모 주석 영상 데이터가 부족하기 때문에 영상 영역에서 종종 저하된다. 본 연구에서는 해양 영상 이해에 초점을 맞춘다. 이는 추가적인 도전 과제를 수반한다. 첫째, 상당한 도메인 전문 지식이 필요하다. 둘째, 영상 VLM은 유익한 이벤트가 일반적으로 희소하고 예측 불가능하며 불균등하게 분포하기 때문에 해양 영상에서 중요한 정보를 국소화하고 해석하는 데 어려움을 겪는다. 이러한 과제를 해결하기 위해, 우리는 MarineEVT라는 최초의 이벤트 중심 해양 영상 이해 데이터셋을 신중하게 구축했다. 이 데이터셋은 해양 이해 및 분석의 여러 차원에 걸친 2만 개의 다중 태스크, 영상 수준 시각적 질의응답 쌍을 포함한다. 한편, MarineEVT를 기반으로 해양 영상 이해를 이벤트 중심 시각적 도구 통합 추론(Event-centric Visual Tool-integrated Reasoning) 과정인 EVT-R1(약칭)으로 분해한다. 여기서 강력한 시각적 도구를 활용하여 모델이 시각적 질문과 인간의 의도에 부합하는 중요한 정보를 국소화하고 해석하도록 유도한다. 효과를 입증하기 위해 다양한 설정에서 EVT-R1을 11개의 SOTA VLM과 비교한다. EVT-R1은 최고 수준의 오픈소스 모델과 상용 모델을 각각 5.22, 11.09만큼 능가한다. MarineEVT와 EVT-R1은 생태학적 발견과 해양 교육의 기반을 마련하며, 해양 역학을 해석하고 생태적 상호작용에 대해 추론하며, 지속 가능한 해양 영상 이해 및 분석을 지원하는 VLM의 개발을 촉진한다.
Face Anti-Spoofing (FAS) is essential for ensuring the security and reliability of facial recognition systems. Most existing FAS methods are formulated as binary classification tasks, providing confidence scores without interpretation. They exhibit limited generalization in out-of-domain scenarios, such as new environments or unseen spoofing types. In this work, we introduce a multimodal large language model (MLLM) framework for FAS, termed Interpretable Face Anti-Spoofing (I-FAS), which transforms the FAS task into an interpretable visual question answering (VQA) paradigm. Specifically, we propose a Spoof-aware Captioning and Filtering (SCF) strategy to generate high-quality captions for FAS images, enriching the model's supervision with natural language interpretations. To mitigate the impact of noisy captions during training, we develop a Lopsided Language Model (L-LM) loss function that separates loss calculations for judgment and interpretation, prioritizing the optimization of the former. Furthermore, to enhance the model's perception of global visual features, we design a Globally Aware Connector (GAC) to align multi-level visual representations with the language model. Extensive experiments on standard and newly devised One to Eleven cross-domain benchmarks, comprising 12 public datasets, demonstrate that our method significantly outperforms state-of-the-art methods.
Multimodal Large Language Models (MLLMs) have achieved significant advancements in tasks like Visual Question Answering (VQA) by leveraging foundational Large Language Models (LLMs). However, their abilities in specific areas such as temporal understanding, which is crucial for comprehending real-world dynamics, remain underexplored. To address this, we propose a challenging evaluation benchmark named TemporalVQA, consisting of two parts: (1) Temporal Order Understanding and (2) Time-lapse Estimation. The first part requires MLLMs to determine the sequence of events by analyzing temporally consecutive video frames. The second part presents image pairs with varying time differences, framed as multiple-choice questions, asking MLLMs to estimate the time-lapse between images with options ranging from seconds to years. Our evaluations of advanced MLLMs, including models like GPT-4o and Gemini-1.5-Pro, reveal significant challenges: GPT-4o achieved only 43.8% average consistent accuracy in temporal order tasks and 70% in time-lapse estimation, with open-source models performing even less effectively. These findings underscore the limitations of current MLLMs in visual temporal understanding and reasoning, highlighting the need for further improvements in their temporal capabilities. Our dataset can be found at https://huggingface.co/datasets/fazliimam/temporal-vqa.
While existing video benchmarks largely consider specialized downstream tasks like retrieval or question-answering (QA), contemporary multimodal AI systems must be capable of well-rounded common-sense reasoning akin to human visual understanding. A critical component of human temporal-visual perception is our ability to identify and cognitively model "things happening", or events. Historically, video benchmark tasks have implicitly tested for this ability (e.g., video captioning, in which models describe visual events with natural language), but they do not consider video event understanding as a task in itself. Recent work has begun to explore video analogues to textual event extraction but consists of competing task definitions and datasets limited to highly specific event types. Therefore, while there is a rich domain of event-centric video research spanning the past 10+ years, it is unclear how video event understanding should be framed and what resources we have to study it. In this paper, we survey 105 video datasets that require event understanding capability, consider how they contribute to the study of robust event understanding in video, and assess proposed video event extraction tasks in the context of this body of research. We propose suggestions informed by this survey for dataset curation and task framing, with an emphasis on the uniquely temporal nature of video events and ambiguity in visual content.
Visually rich documents encode relevance through language, layout, structured visual elements, and corpus context, yet retrieval is typically evaluated by one-shot query--page matching. Agentic-search benchmarks usually score downstream question answering or report generation, leaving document ranking under iterative evidence acquisition underexplored. We introduce VisDocAgentBench, a closed-corpus benchmark comparing static and agentic retrieval under a shared ranked-output contract. It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures. Relation-preserving construction yields semantic, relational, and visual queries, followed by full-document review and hard-negative validation. A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context. Agents recover much of this loss, but planner choice and retrieval representation remain decisive. Every planner performs better with visual retrieval, whose best R@1 reaches 67.50% versus 37.50% for OCR-text. Ablations identify iterative search and page inspection as consequential capabilities, and providing the complete support context improves ranking on both routes. Trace analysis localizes the remaining losses to target discovery, candidate examination, and evidence-role integration. These findings motivate retrieval agents that combine modality-preserving discovery with evidence-directed verification.
To increase the generalization capability of VQA systems, many recent studies have tried to de-bias spurious language or vision associations that shortcut the question or image to the answer. Despite these efforts, the literature fails to address the confounding effect of vision and language simultaneously. As a result, when they reduce bias learned from one modality, they usually increase bias from the other. In this paper, we first model a confounding effect that causes language and vision bias simultaneously, then propose a counterfactual inference to remove the influence of this effect. The model trained in this strategy can concurrently and efficiently reduce vision and language bias. To the best of our knowledge, this is the first work to reduce biases resulting from confounding effects of vision and language in VQA, leveraging causal explain-away relations. We accompany our method with an explain-away strategy, pushing the accuracy of the questions with numerical answers results compared to existing methods that have been an open problem. The proposed method outperforms the state-of-the-art methods in VQA-CP v2 datasets.
Answering questions that require reading texts in an image is challenging for current models. One key difficulty of this task is that rare, polysemous, and ambiguous words frequently appear in images, e.g., names of places, products, and sports teams. To overcome this difficulty, only resorting to pre-trained word embedding models is far from enough. A desired model should utilize the rich information in multiple modalities of the image to help understand the meaning of scene texts, e.g., the prominent text on a bottle is most likely to be the brand. Following this idea, we propose a novel VQA approach, Multi-Modal Graph Neural Network (MM-GNN). It first represents an image as a graph consisting of three sub-graphs, depicting visual, semantic, and numeric modalities respectively. Then, we introduce three aggregators which guide the message passing from one graph to another to utilize the contexts in various modalities, so as to refine the features of nodes. The updated nodes have better features for the downstream question answering module. Experimental evaluations show that our MM-GNN represents the scene texts better and obviously facilitates the performances on two VQA tasks that require reading scene texts.
Language has become a prominent modality in computer vision with the rise of LLMs. Despite supporting long context-lengths, their effectiveness in handling long-term information gradually declines with input length. This becomes critical, especially in applications such as long-form video understanding. In this paper, we introduce a Language Repository (LangRepo) for LLMs, that maintains concise and structured information as an interpretable (i.e., all-textual) representation. Our repository is updated iteratively based on multi-scale video chunks. We introduce write and read operations that focus on pruning redundancies in text, and extracting information at various temporal scales. The proposed framework is evaluated on zero-shot visual question-answering benchmarks including EgoSchema, NExT-QA, IntentQA and NExT-GQA, showing state-of-the-art performance at its scale. Our code is available at https://github.com/kkahatapitiya/LangRepo.
Large vision-and-language models (VLMs) trained to match images with text on large-scale datasets of image-text pairs have shown impressive generalization ability on several vision and language tasks. Several recent works, however, showed that these models lack fine-grained understanding, such as the ability to count and recognize verbs, attributes, or relationships. The focus of this work is to study the understanding of spatial relations. This has been tackled previously using image-text matching (e.g., Visual Spatial Reasoning benchmark) or visual question answering (e.g., GQA or VQAv2), both showing poor performance and a large gap compared to human performance. In this work, we show qualitatively (using explainability tools) and quantitatively (using object detectors) that the poor object localization "grounding" ability of the models is a contributing factor to the poor image-text matching performance. We propose an alternative fine-grained, compositional approach for recognizing and ranking spatial clauses that combines the evidence from grounding noun phrases corresponding to objects and their locations to compute the final rank of the spatial clause. We demonstrate the approach on representative VLMs (such as LXMERT, GPV, and MDETR) and compare and highlight their abilities to reason about spatial relationships.
Medical large vision-language models (LVLMs) have demonstrated promising performance across various single-image question answering (QA) benchmarks, yet their capability in processing multi-image clinical scenarios remains underexplored. Unlike single image based tasks, medical tasks involving multiple images often demand sophisticated visual understanding capabilities, such as temporal reasoning and cross-modal analysis, which are poorly supported by current medical LVLMs. To bridge this critical gap, we present the Med-MIM instruction dataset, comprising 83.2K medical multi-image QA pairs that span four types of multi-image visual abilities (temporal understanding, reasoning, comparison, co-reference). Using this dataset, we fine-tune Mantis and LLaVA-Med, resulting in two specialized medical VLMs: MIM-LLaVA-Med and Med-Mantis, both optimized for multi-image analysis. Additionally, we develop the Med-MIM benchmark to comprehensively evaluate the medical multi-image understanding capabilities of LVLMs. We assess eight popular LVLMs, including our two models, on the Med-MIM benchmark. Experimental results show that both Med-Mantis and MIM-LLaVA-Med achieve superior performance on the held-in and held-out subsets of the Med-MIM benchmark, demonstrating that the Med-MIM instruction dataset effectively enhances LVLMs' multi-image understanding capabilities in the medical domain.
검색-증강 생성(RAG)은 지식 집약적 질의응답에서 필수적인 요소가 되었으나, 검색된 긴 문맥을 처리하는 데 드는 막대한 연산 비용으로 인해 RAG 파이프라인의 확장(scaling)은 여전히 어렵습니다. 기존의 압축 방식들은 근본적인 절충(trade-off) 관계에 직면해 있습니다. 하드 압축(hard compression) 방법은 질의 인식(query-aware) 방식으로 온라인에서 동작하지만 압축률이 완만하고 일반적으로 생성 모델의 미세조정을 요구하는 반면, 소프트 압축(soft compression) 방법은 더 높은 압축률을 달성하지만 입력 질의와 완전히 무관한 고비용의 오프라인 인코딩에 의존합니다. 이러한 격차를 해소하기 위해, 본 연구에서는 입력 질의에 맞춰 검색된 문서를 간결한 시각적 표현으로 압축하는 새로운 프레임워크인 RAGOCR을 소개합니다. 압축률과 정보 충실도 사이의 균형을 더욱 맞추기 위해, 각 문서의 추정된 관련성과 복잡성에 따라 시각적 세밀도를 적응적으로 할당하는 질의 인식 동적 해상도 메커니즘을 도입합니다. 즉, 관련성이 높은 구절은 세밀한 세부 정보를 보존하기 위해 더 높은 해상도로 렌더링되는 반면, 주변 문서는 더 낮은 해상도로 공격적으로 압축됩니다. MedOmniKB 검색 코퍼스를 사용한 5개의 질의응답(QA) 벤치마크 실험에 따르면, RAGOCR은 입력 토큰을 8분의 1만 요구하면서도 단순(naive) RAG 대비 정확도를 15% 이상 능가했으며, 다양한 검색 깊이에 걸쳐 하드 및 소프트 압축 베이스라인 모두를 일관되게 능가함을 보여줍니다.
Research in scene graph generation has quickly gained traction in the past few years because of its potential to help in downstream tasks like visual question answering, image captioning, etc. Many interesting approaches have been proposed to tackle this problem. Most of these works have a pre-trained object detection model as a preliminary feature extractor. Therefore, getting object bounding box proposals from the object detection model is relatively cheaper. We take advantage of this ready availability of bounding box annotations produced by the pre-trained detector. We propose a set of three novel yet simple self-supervision tasks and train them as auxiliary multi-tasks to the main model. While comparing, we train the base-model from scratch with these self-supervision tasks, we achieve state-of-the-art results in all the metrics and recall settings. We also resolve some of the confusion between two types of relationships: geometric and possessive, by training the model with the proposed self-supervision losses. We use the benchmark dataset, Visual Genome to conduct our experiments and show our results.
Multimodal Knowledge Graphs (MKGs), which organize visual-text factual knowledge, have recently been successfully applied to tasks such as information retrieval, question answering, and recommendation system. Since most MKGs are far from complete, extensive knowledge graph completion studies have been proposed focusing on the multimodal entity, relation extraction and link prediction. However, different tasks and modalities require changes to the model architecture, and not all images/objects are relevant to text input, which hinders the applicability to diverse real-world scenarios. In this paper, we propose a hybrid transformer with multi-level fusion to address those issues. Specifically, we leverage a hybrid transformer architecture with unified input-output for diverse multimodal knowledge graph completion tasks. Moreover, we propose multi-level fusion, which integrates visual and text representation via coarse-grained prefix-guided interaction and fine-grained correlation-aware fusion modules. We conduct extensive experiments to validate that our MKGformer can obtain SOTA performance on four datasets of multimodal link prediction, multimodal RE, and multimodal NER. Code is available in https://github.com/zjunlp/MKGformer.