From Shallow to Deep: Compositional Reasoning over Graphs for Visual Question Answering
원문 초록
In order to achieve a general visual question answering (VQA) system, it is essential to learn to answer deeper questions that require compositional reasoning on the image and external knowledge. Meanwhile, the reasoning process should be explicit and explainable to understand the working mechanism of the model. It is effortless for human but challenging for machines. In this paper, we propose a Hierarchical Graph Neural Module Network (HGNMN) that reasons over multi-layer graphs with neural modules to address the above issues. Specifically, we first encode the image by multi-layer graphs from the visual, semantic and commonsense views since the clues that support the answer may exist in different modalities. Our model consists of several well-designed neural modules that perform specific functions over graphs, which can be used to conduct multi-step reasoning within and between different graphs. Compared to existing modular networks, we extend visual reasoning from one graph to more graphs. We can explicitly trace the reasoning process according to module weights and graph attentions. Experiments show that our model not only achieves state-of-the-art performance on the CRIC dataset but also obtains explicit and explainable reasoning procedures.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Describe Anything Model for Visual Question Answering on Text-rich Images
DrishtiKon: Multi-Granular Visual Grounding for Text-Rich Document Images
HRIBench: Benchmarking Vision-Language Models for Real-Time Human Perception in Human-Robot Interaction
Adapting Lightweight Vision Language Models for Radiological Visual Question Answering
SlotPi: Physics-informed Object-centric Reasoning Models
Outside Knowledge Conversational Video (OKCV) Dataset -- Dialoguing over Videos
토론
이 논문과 연결된 포럼 글이 아직 없습니다.