HAAV: Hierarchical Aggregation of Augmented Views for Image Captioning
원문 초록
A great deal of progress has been made in image captioning, driven by research into how to encode the image using pre-trained models. This includes visual encodings (e.g. image grid features or detected objects) and more recently textual encodings (e.g. image tags or text descriptions of image regions). As more advanced encodings are available and incorporated, it is natural to ask: how to efficiently and effectively leverage the heterogeneous set of encodings? In this paper, we propose to regard the encodings as augmented views of the input image. The image captioning model encodes each view independently with a shared encoder efficiently, and a contrastive loss is incorporated across the encoded views in a novel way to improve their representation quality and the model's data efficiency. Our proposed hierarchical decoder then adaptively weighs the encoded views according to their effectiveness for caption generation by first aggregating within each view at the token level, and then across views at the view level. We demonstrate significant performance improvements of +5.6% CIDEr on MS-COCO and +12.9% CIDEr on Flickr30k compared to state of the arts, and conduct rigorous analyses to demonstrate the importance of each part of our design.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning
Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation
Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learning
MedBLIP: Fine-tuning BLIP for Medical Image Captioning
Representation Learning through Multimodal Attention and Time-Sync Comments for Affective Video Content Analysis
Global Object Proposals for Improving Multi-Sentence Video Descriptions
토론
이 논문과 연결된 포럼 글이 아직 없습니다.