MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization
원문 초록
Generating music that aligns with the visual content of a video has been a challenging task, as it requires a deep understanding of visual semantics and involves generating music whose melody, rhythm, and dynamics harmonize with the visual narratives. This paper presents MuVi, a novel framework that effectively addresses these challenges to enhance the cohesion and immersive experience of audio-visual content. MuVi analyzes video content through a specially designed visual adaptor to extract contextually and temporally relevant features. These features are used to generate music that not only matches the video's mood and theme but also its rhythm and pacing. We also introduce a contrastive music-visual pre-training scheme to ensure synchronization, based on the periodicity nature of music phrases. In addition, we demonstrate that our flow-matching-based music generator has in-context learning ability, allowing us to control the style and genre of the generated music. Experimental results show that MuVi demonstrates superior performance in both audio quality and temporal synchronization. The generated music video samples are available at https://muvi-v2m.github.io.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
From Generality to Mastery: Composer-Style Symbolic Music Generation via Large-Scale Pre-training
Exploring Adapter Design Tradeoffs for Low Resource Music Generation
Let Your Video Listen to Your Music!
Deep learning for music generation. Four approaches and their comparative evaluation
Assay2Mol: large language model-based drug design using BioAssay context
WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modeling
토론
이 논문과 연결된 포럼 글이 아직 없습니다.