Video Referring Expression Comprehension via Transformer with Content-conditioned Query
원문 초록
Video Referring Expression Comprehension (REC) aims to localize a target object in videos based on the queried natural language. Recent improvements in video REC have been made using Transformer-based methods with learnable queries. However, we contend that this naive query design is not ideal given the open-world nature of video REC brought by text supervision. With numerous potential semantic categories, relying on only a few slow-updated queries is insufficient to characterize them. Our solution to this problem is to create dynamic queries that are conditioned on both the input video and language to model the diverse objects referred to. Specifically, we place a fixed number of learnable bounding boxes throughout the frame and use corresponding region features to provide prior information. Also, we noticed that current query features overlook the importance of cross-modal alignment. To address this, we align specific phrases in the sentence with semantically relevant visual areas, annotating them in existing video datasets (VID-Sentence and VidSTG). By incorporating these two designs, our proposed model (called ConFormer) outperforms other models on widely benchmarked datasets. For example, in the testing split of VID-Sentence dataset, ConFormer achieves 8.75% absolute improvement on [email omitted] compared to the previous state-of-the-art model.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models
WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation
DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learning
Referring Expression Instance Retrieval and A Strong End-to-End Baseline
Synthetic Visual Genome
Deformable Attentive Visual Enhancement for Referring Segmentation Using Vision-Language Model
토론
이 논문과 연결된 포럼 글이 아직 없습니다.