A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehension
원문 초록
Referring expression comprehension aims to localize the object instance described by a natural language expression. Current referring expression methods have achieved good performance. However, none of them is able to achieve real-time inference without accuracy drop. The reason for the relatively slow inference speed is that these methods artificially split the referring expression comprehension into two sequential stages including proposal generation and proposal ranking. It does not exactly conform to the habit of human cognition. To this end, we propose a novel Realtime Cross-modality Correlation Filtering method (RCCF). RCCF reformulates the referring expression comprehension as a correlation filtering process. The expression is first mapped from the language domain to the visual domain and then treated as a template (kernel) to perform correlation filtering on the image feature map. The peak value in the correlation heatmap indicates the center points of the target box. In addition, RCCF also regresses a 2-D object size and 2-D offset. The center point coordinates, object size and center point offset together to form the target bounding box. Our method runs at 40 FPS while achieving leading performance in RefClef, RefCOCO, RefCOCO+ and RefCOCOg benchmarks. In the challenging RefClef dataset, our methods almost double the state-of-the-art performance (34.70% increased to 63.79%). We hope this work can arouse more attention and studies to the new cross-modality correlation filtering framework as well as the one-stage framework for referring expression comprehension.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models
RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenes
Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints
Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models
Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE
FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension
토론
이 논문과 연결된 포럼 글이 아직 없습니다.