Visually Informed Binaural Audio Generation without Binaural Audios
원문 초록
Stereophonic audio, especially binaural audio, plays an essential role in immersive viewing environments. Recent research has explored generating visually guided stereophonic audios supervised by multi-channel audio collections. However, due to the requirement of professional recording devices, existing datasets are limited in scale and variety, which impedes the generalization of supervised methods in real-world scenarios. In this work, we propose PseudoBinaural, an effective pipeline that is free of binaural recordings. The key insight is to carefully build pseudo visual-stereo pairs with mono data for training. Specifically, we leverage spherical harmonic decomposition and head-related impulse response (HRIR) to identify the relationship between spatial locations and received binaural audios. Then in the visual modality, corresponding visual cues of the mono data are manually placed at sound source positions to form the pairs. Compared to fully-supervised paradigms, our binaural-recording-free pipeline shows great stability in cross-dataset evaluation and achieves comparable performance under subjective preference. Moreover, combined with binaural recordings, our method is able to further boost the performance of binaural audio generation under supervised settings.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
BemaGANv2: A Tutorial and Comparative Survey of GAN-based Vocoders for Long-Term Audio Generation
XMAD-Bench: Cross-Domain Multilingual Audio Deepfake Benchmark
Conditional Diffusion Models with Classifier-Free Gibbs-like Guidance
Fast Text-to-Audio Generation with Adversarial Post-Training
OmniAudio: Generating Spatial Audio from 360-Degree Video
Detect All-Type Deepfake Audio: Wavelet Prompt Tuning for Enhanced Auditory Perception
토론
이 논문과 연결된 포럼 글이 아직 없습니다.