Global2Local: A Joint-Hierarchical Attention for Video Captioning
원문 초록
Recently, automatic video captioning has attracted increasing attention, where the core challenge lies in capturing the key semantic items, like objects and actions as well as their spatial-temporal correlations from the redundant frames and semantic content. To this end, existing works select either the key video clips in a global level~(across multi frames), or key regions within each frame, which, however, neglect the hierarchical order, i.e., key frames first and key regions latter. In this paper, we propose a novel joint-hierarchical attention model for video captioning, which embeds the key clips, the key frames and the key regions jointly into the captioning model in a hierarchical manner. Such a joint-hierarchical attention model first conducts a global selection to identify key frames, followed by a Gumbel sampling operation to identify further key regions based on the key frames, achieving an accurate global-to-local feature representation to guide the captioning. Extensive quantitative evaluations on two public benchmark datasets MSVD and MSR-VTT demonstrates the superiority of the proposed method over the state-of-the-art methods.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarks
video-SALMONN 2: Captioning-Enhanced Audio-Visual Large Language Models
FLASH: Latent-Aware Semi-Autoregressive Speculative Decoding for Multimodal Tasks
FocusedAD: Character-centric Movie Audio Description
Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Prompting
Pretrained Image-Text Models are Secretly Video Captioners
토론
이 논문과 연결된 포럼 글이 아직 없습니다.