SoundBrush: Sound as a Brush for Visual Scene Editing
원문 초록
We propose SoundBrush, a model that uses sound as a brush to edit and manipulate visual scenes. We extend the generative capabilities of the Latent Diffusion Model (LDM) to incorporate audio information for editing visual scenes. Inspired by existing image-editing works, we frame this task as a supervised learning problem and leverage various off-the-shelf models to construct a sound-paired visual scene dataset for training. This richly generated dataset enables SoundBrush to learn to map audio features into the textual space of the LDM, allowing for visual scene editing guided by diverse in-the-wild sound. Unlike existing methods, SoundBrush can accurately manipulate the overall scenery or even insert sounding objects to best match the audio inputs while preserving the original content. Furthermore, by integrating with novel view synthesis techniques, our framework can be extended to edit 3D scenes, facilitating sound-driven 3D scene manipulation. Demos are available at https://soundbrush.github.io/.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Refine Any Object in Any Scene
UnMix-NeRF: Spectral Unmixing Meets Neural Radiance Fields
Self-Supervised Multimodal NeRF for Autonomous Driving
R3eVision: A Survey on Robust Rendering, Restoration, and Enhancement for 3D Low-Level Vision
3DGS-IEval-15K: A Large-scale Image Quality Evaluation Database for 3D Gaussian-Splatting
TextureSplat: Per-Primitive Texture Mapping for Reflective Gaussian Splatting
토론
이 논문과 연결된 포럼 글이 아직 없습니다.