Jointly Visual- and Semantic-Aware Graph Memory Networks for Temporal Sentence Localization in Videos
원문 초록
Temporal sentence localization in videos (TSLV) aims to retrieve the most interested segment in an untrimmed video according to a given sentence query. However, almost of existing TSLV approaches suffer from the same limitations: (1) They only focus on either frame-level or object-level visual representation learning and corresponding correlation reasoning, but fail to integrate them both; (2) They neglect to leverage the rich semantic contexts to further benefit the query reasoning. To address these issues, in this paper, we propose a novel Hierarchical Visual- and Semantic-Aware Reasoning Network (HVSARN), which enables both visual- and semantic-aware query reasoning from object-level to frame-level. Specifically, we present a new graph memory mechanism to perform visual-semantic query reasoning: For visual reasoning, we design a visual graph memory to leverage visual information of video; For semantic reasoning, a semantic graph memory is also introduced to explicitly leverage semantic knowledge contained in the classes and attributes of video objects, and perform correlation reasoning in the semantic space. Experiments on three datasets demonstrate that our HVSARN achieves a new state-of-the-art performance.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MiCo: Multi-image Contrast for Reinforcement Visual Reasoning
DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learning
Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
Representation Learning through Multimodal Attention and Time-Sync Comments for Affective Video Content Analysis
DIBERT: Dependency Injected Bidirectional Encoder Representations from Transformers
W2VV++: Fully Deep Learning for Ad-hoc Video Search
토론
이 논문과 연결된 포럼 글이 아직 없습니다.