Synthesizing Audio from Silent Video using Sequence to Sequence Modeling
원문 초록
Generating audio from a video's visual context has multiple practical applications in improving how we interact with audio-visual media - for example, enhancing CCTV footage analysis, restoring historical videos (e.g., silent movies), and improving video generation models. We propose a novel method to generate audio from video using a sequence-to-sequence model, improving on prior work that used CNNs and WaveNet and faced sound diversity and generalization challenges. Our approach employs a 3D Vector Quantized Variational Autoencoder (VQ-VAE) to capture the video's spatial and temporal structures, decoding with a custom audio decoder for a broader range of sounds. Trained on the Youtube8M dataset segment, focusing on specific domains, our model aims to enhance applications like CCTV footage analysis, silent movie restoration, and video generation models.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
$I^{2}$-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting
Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation
Sparse Regression Codes exploit Multi-User Diversity without CSI
A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Quality
DEU-Net: Dual-Encoder U-Net for Automated Skin Lesion Segmentation
Deep Encoder-Decoder Networks for Classification of Hyperspectral and LiDAR Data
토론
이 논문과 연결된 포럼 글이 아직 없습니다.