Visual Storytelling with Question-Answer Plans
원문 초록
Visual storytelling aims to generate compelling narratives from image sequences. Existing models often focus on enhancing the representation of the image sequence, e.g., with external knowledge sources or advanced graph structures. Despite recent progress, the stories are often repetitive, illogical, and lacking in detail. To mitigate these issues, we present a novel framework which integrates visual representations with pretrained language models and planning. Our model translates the image sequence into a visual prefix, a sequence of continuous embeddings which language models can interpret. It also leverages a sequence of question-answer pairs as a blueprint plan for selecting salient visual concepts and determining how they should be assembled into a narrative. Automatic and human evaluation on the VIST benchmark (Huang et al., 2016) demonstrates that blueprint-based models generate stories that are more coherent, interesting, and natural compared to competitive baselines and state-of-the-art systems.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Consistent Story Generation with Asymmetry Zigzag Sampling
StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation
FLIP Reasoning Challenge
FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations
Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
Alfie: Democratising RGBA Image Generation With No $$$
토론
이 논문과 연결된 포럼 글이 아직 없습니다.