Uncertainty-Aware Image Captioning
원문 초록
It is well believed that the higher uncertainty in a word of the caption, the more inter-correlated context information is required to determine it. However, current image captioning methods usually consider the generation of all words in a sentence sequentially and equally. In this paper, we propose an uncertainty-aware image captioning framework, which parallelly and iteratively operates insertion of discontinuous candidate words between existing words from easy to difficult until converged. We hypothesize that high-uncertainty words in a sentence need more prior information to make a correct decision and should be produced at a later stage. The resulting non-autoregressive hierarchy makes the caption generation explainable and intuitive. Specifically, we utilize an image-conditioned bag-of-word model to measure the word uncertainty and apply a dynamic programming algorithm to construct the training pairs. During inference, we devise an uncertainty-adaptive parallel beam search technique that yields an empirically logarithmic time complexity. Extensive experiments on the MS COCO benchmark reveal that our approach outperforms the strong baseline and related methods on both captioning quality as well as decoding speed.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Global Object Proposals for Improving Multi-Sentence Video Descriptions
Document-Level Text Generation with Minimum Bayes Risk Decoding using Optimal Transport
Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation
Group-based Distinctive Image Captioning with Memory Difference Encoding and Attention
Representation Learning through Multimodal Attention and Time-Sync Comments for Affective Video Content Analysis
Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression
토론
이 논문과 연결된 포럼 글이 아직 없습니다.