MUTATT: Visual-Textual Mutual Guidance for Referring Expression Comprehension
원문 초록
Referring expression comprehension (REC) aims to localize a text-related region in a given image by a referring expression in natural language. Existing methods focus on how to build convincing visual and language representations independently, which may significantly isolate visual and language information. In this paper, we argue that for REC the referring expression and the target region are semantically correlated and subject, location and relationship consistency exist between vision and language.On top of this, we propose a novel approach called MutAtt to construct mutual guidance between vision and language, which treat vision and language equally thus yield compact information matching. Specifically, for each module of subject, location and relationship, MutAtt builds two kinds of attention-based mutual guidance strategies. One strategy is to generate vision-guided language embedding for the sake of matching relevant visual feature. The other reversely generates language-guided visual feature to match relevant language embedding. This mutual guidance strategy can effectively guarantees the vision-language consistency in three modules. Experiments on three popular REC datasets demonstrate that the proposed approach outperforms the current state-of-the-art methods.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models
RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenes
Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints
Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models
Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE
FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension
토론
이 논문과 연결된 포럼 글이 아직 없습니다.