DECap: Towards Generalized Explicit Caption Editing via Diffusion Mechanism
원문 초록
Explicit Caption Editing (ECE) -- refining reference image captions through a sequence of explicit edit operations (e.g., KEEP, DETELE) -- has raised significant attention due to its explainable and human-like nature. After training with carefully designed reference and ground-truth caption pairs, state-of-the-art ECE models exhibit limited generalization ability beyond the original training data distribution, i.e., they are tailored to refine content details only in in-domain samples but fail to correct errors in out-of-domain samples. To this end, we propose a new Diffusion-based Explicit Caption editing method: DECap. Specifically, we reformulate the ECE task as a denoising process under the diffusion mechanism, and introduce innovative edit-based noising and denoising processes. Thanks to this design, the noising process can help to eliminate the need for meticulous paired data selection by directly introducing word-level noises for training, learning diverse distribution over input reference caption. The denoising process involves the explicit predictions of edit operations and corresponding content words, refining reference captions through iterative step-wise editing. To further efficiently implement our diffusion process and improve the inference speed, DECap discards the prevalent multi-stage design and directly generates edit operations and content words simultaneously. Extensive ablations have demonstrated the strong generalization ability of DECap in various scenarios. More interestingly, it even shows great potential in improving the quality and controllability of caption generation.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation
Global Object Proposals for Improving Multi-Sentence Video Descriptions
Refining Few-Step Text-to-Multiview Diffusion via Reinforcement Learning
On Designing Diffusion Autoencoders for Efficient Generation and Representation Learning
ScoreAdv: Score-based Targeted Generation of Natural Adversarial Examples via Diffusion Models
LangMamba: A Language-driven Mamba Framework for Low-dose CT Denoising with Vision-language Models
토론
이 논문과 연결된 포럼 글이 아직 없습니다.