Video Question Answering Using CLIP-Guided Visual-Text Attention
원문 초록
Cross-modal learning of video and text plays a key role in Video Question Answering (VideoQA). In this paper, we propose a visual-text attention mechanism to utilize the Contrastive Language-Image Pre-training (CLIP) trained on lots of general domain language-image pairs to guide the cross-modal learning for VideoQA. Specifically, we first extract video features using a TimeSformer and text features using a BERT from the target application domain, and utilize CLIP to extract a pair of visual-text features from the general-knowledge domain through the domain-specific learning. We then propose a Cross-domain Learning to extract the attention information between visual and linguistic features across the target domain and general domain. The set of CLIP-guided visual-text features are integrated to predict the answer. The proposed method is evaluated on MSVD-QA and MSRVTT-QA datasets, and outperforms state-of-the-art methods.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Decoupled Seg Tokens Make Stronger Reasoning Video Segmenter and Grounder
LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs
video-SALMONN 2: Captioning-Enhanced Audio-Visual Large Language Models
CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
Looking Beyond Visible Cues: Implicit Video Question Answering via Dual-Clue Reasoning
토론
이 논문과 연결된 포럼 글이 아직 없습니다.