Graph-Structured Referring Expression Reasoning in The Wild
원문 초록
Grounding referring expressions aims to locate in an image an object referred to by a natural language expression. The linguistic structure of a referring expression provides a layout of reasoning over the visual contents, and it is often crucial to align and jointly understand the image and the referring expression. In this paper, we propose a scene graph guided modular network (SGMN), which performs reasoning over a semantic graph and a scene graph with neural modules under the guidance of the linguistic structure of the expression. In particular, we model the image as a structured semantic graph, and parse the expression into a language scene graph. The language scene graph not only decodes the linguistic structure of the expression, but also has a consistent representation with the image semantic graph. In addition to exploring structured solutions to grounding referring expressions, we also propose Ref-Reasoning, a large-scale real-world dataset for structured referring expression reasoning. We automatically generate referring expressions over the scene graphs of images using diverse expression templates and functional programs. This dataset is equipped with real-world visual contents as well as semantically rich expressions with different reasoning layouts. Experimental results show that our SGMN not only significantly outperforms existing state-of-the-art algorithms on the new Ref-Reasoning dataset, but also surpasses state-of-the-art structured methods on commonly used benchmark datasets. It can also provide interpretable visual evidences of reasoning. Data and code are available at https://github.com/sibeiyang/sgmn
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
What is the Right Embedding Space for Contrastive Learning in REC?
Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models
TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models
RemoteSAM: Towards Segment Anything for Earth Observation
Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation
4th PVUW MeViS 3rd Place Report: Sa2VA
토론
이 논문과 연결된 포럼 글이 아직 없습니다.