DiffCap: Exploring Continuous Diffusion on Image Captioning
원문 초록
Current image captioning works usually focus on generating descriptions in an autoregressive manner. However, there are limited works that focus on generating descriptions non-autoregressively, which brings more decoding diversity. Inspired by the success of diffusion models on generating natural-looking images, we propose a novel method DiffCap to apply continuous diffusions on image captioning. Unlike image generation where the output is fixed-size and continuous, image description length varies with discrete tokens. Our method transforms discrete tokens in a natural way and applies continuous diffusion on them to successfully fuse extracted image features for diffusion caption generation. Our experiments on COCO dataset demonstrate that our method uses a much simpler structure to achieve comparable results to the previous non-autoregressive works. Apart from quality, an intriguing property of DiffCap is its high diversity during generation, which is missing from many autoregressive models. We believe our method on fusing multimodal features in diffusion language generation will inspire more researches on multimodal language generation tasks for its simplicity and decoding flexibility.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Synthesizing Reality: Leveraging the Generative AI-Powered Platform Midjourney for Construction Worker Detection
AdaptaGen: Domain-Specific Image Generation through Hierarchical Semantic Optimization Framework
Beyond Blur: A Fluid Perspective on Generative Diffusion Models
AI's Blind Spots: Geographic Knowledge and Diversity Deficit in Generated Urban Scenario
SPARKE: Scalable Prompt-Aware Diversity Guidance in Diffusion Models via RKE Score
Noise Consistency Regularization for Improved Subject-Driven Image Synthesis
토론
이 논문과 연결된 포럼 글이 아직 없습니다.