Video Captioning Using Weak Annotation
원문 초록
Video captioning has shown impressive progress in recent years. One key reason of the performance improvements made by existing methods lie in massive paired video-sentence data, but collecting such strong annotation, i.e., high-quality sentences, is time-consuming and laborious. It is the fact that there now exist an amazing number of videos with weak annotation that only contains semantic concepts such as actions and objects. In this paper, we investigate using weak annotation instead of strong annotation to train a video captioning model. To this end, we propose a progressive visual reasoning method that progressively generates fine sentences from weak annotations by inferring more semantic concepts and their dependency relationships for video captioning. To model concept relationships, we use dependency trees that are spanned by exploiting external knowledge from large sentence corpora. Through traversing the dependency trees, the sentences are generated to train the captioning model. Accordingly, we develop an iterative refinement algorithm that refines sentences via spanning dependency trees and fine-tunes the captioning model using the refined sentences in an alternative training manner. Experimental results demonstrate that our method using weak annotation is very competitive to the state-of-the-art methods using strong annotation.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Show, Tell and Summarize: Dense Video Captioning Using Visual Cue Aided Sentence Summarization
Dense Video Captioning using Graph-based Sentence Summarization
Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
Describe Anything: Detailed Localized Image and Video Captioning
Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression
Mitigating Object Hallucinations via Sentence-Level Early Intervention
토론
이 논문과 연결된 포럼 글이 아직 없습니다.