Knowledge-enriched Attention Network with Group-wise Semantic for Visual Storytelling
원문 초록
As a technically challenging topic, visual storytelling aims at generating an imaginary and coherent story with narrative multi-sentences from a group of relevant images. Existing methods often generate direct and rigid descriptions of apparent image-based contents, because they are not capable of exploring implicit information beyond images. Hence, these schemes could not capture consistent dependencies from holistic representation, impairing the generation of reasonable and fluent story. To address these problems, a novel knowledge-enriched attention network with group-wise semantic model is proposed. Three main novel components are designed and supported by substantial experiments to reveal practical advantages. First, a knowledge-enriched attention network is designed to extract implicit concepts from external knowledge system, and these concepts are followed by a cascade cross-modal attention mechanism to characterize imaginative and concrete representations. Second, a group-wise semantic module with second-order pooling is developed to explore the globally consistent guidance. Third, a unified one-stage story generation model with encoder-decoder structure is proposed to simultaneously train and infer the knowledge-enriched attention network, group-wise semantic module and multi-modal story generation decoder in an end-to-end fashion. Substantial experiments on the popular Visual Storytelling dataset with both objective and subjective evaluation metrics demonstrate the superior performance of the proposed scheme as compared with other state-of-the-art methods.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Consistent Story Generation with Asymmetry Zigzag Sampling
StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation
Action2Dialogue: Generating Character-Centric Narratives from Scene-Level Prompts
Representation Learning through Multimodal Attention and Time-Sync Comments for Affective Video Content Analysis
Hierarchical memory decoder for visual narrating
Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression
토론
이 논문과 연결된 포럼 글이 아직 없습니다.