arXiv:2111.12727

Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets

Marcella Cornia, Lorenzo Baraldi, Giuseppe Fiameni, Rita Cucchiara

제출 2021-11-24

원문 초록

This paper addresses the task of generating fluent descriptions by training on a non-uniform combination of data sources, containing both human-annotated and web-collected captions. Large-scale datasets with noisy image-text pairs, indeed, provide a sub-optimal source of supervision because of their low-quality descriptive style, while human-annotated datasets are cleaner but smaller in scale. To get the best of both worlds, we propose to leverage and separate semantics and descriptive style through the incorporation of a style token and keywords extracted through a retrieval component. The proposed model avoids the need of object detectors, is trained with a single objective of prompt language modeling, and can replicate the style of human-collected captions while training on sources with different input styles. Experimentally, the model shows a strong capability of recognizing real-world concepts and producing high-quality captions. Extensive experiments are performed on different image captioning datasets, including CC3M, nocaps, and the competitive COCO dataset, where our model consistently outperforms baselines and state-of-the-art approaches.

함께 살펴볼 논문

공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.

arXiv:2507.06204공통 태그 Retrieval, Language Modeling, Language Modelling공식 코드 · Papers with Code

Differential Mamba

토론

이 논문과 연결된 포럼 글이 아직 없습니다.