Video Soundtrack Generation by Aligning Emotions and Temporal Boundaries
원문 초록
We introduce EMSYNC, a video-based symbolic music generation model that aligns music with a video's emotional content and temporal boundaries. It follows a two-stage framework, where a pretrained video emotion classifier extracts emotional features, and a conditional music generator produces MIDI sequences guided by both emotional and temporal cues. We introduce boundary offsets, a novel temporal conditioning mechanism that enables the model to anticipate and align musical chords with scene cuts. Unlike existing models, our approach retains event-based encoding, ensuring fine-grained timing control and expressive musical nuances. We also propose a mapping scheme to bridge the video emotion classifier, which produces discrete emotion categories, with the emotion-conditioned MIDI generator, which operates on continuous-valued valence-arousal inputs. In subjective listening tests, EMSYNC outperforms state-of-the-art models across all subjective metrics, for music theory-aware participants as well as the general listeners.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modeling
TOMI: Transforming and Organizing Music Ideas for Multi-Track Compositions with Full-Song Structure
From Generality to Mastery: Composer-Style Symbolic Music Generation via Large-Scale Pre-training
Double Entendre: Robust Audio-Based AI-Generated Lyrics Detection via Multi-View Fusion
Personalizable Long-Context Symbolic Music Infilling with MIDI-RWKV
LeVo: High-Quality Song Generation with Multi-Preference Alignment
토론
이 논문과 연결된 포럼 글이 아직 없습니다.