Video ControlNet: Towards Temporally Consistent Synthetic-to-Real Video Translation Using Conditional Image Diffusion Models
원문 초록
In this study, we present an efficient and effective approach for achieving temporally consistent synthetic-to-real video translation in videos of varying lengths. Our method leverages off-the-shelf conditional image diffusion models, allowing us to perform multiple synthetic-to-real image generations in parallel. By utilizing the available optical flow information from the synthetic videos, our approach seamlessly enforces temporal consistency among corresponding pixels across frames. This is achieved through joint noise optimization, effectively minimizing spatial and temporal discrepancies. To the best of our knowledge, our proposed method is the first to accomplish diverse and temporally consistent synthetic-to-real video translation using conditional image diffusion models. Furthermore, our approach does not require any training or fine-tuning of the diffusion models. Extensive experiments conducted on various benchmarks for synthetic-to-real video translation demonstrate the effectiveness of our approach, both quantitatively and qualitatively. Finally, we show that our method outperforms other baseline methods in terms of both temporal consistency and visual quality.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MonoMobility: Zero-Shot 3D Mobility Analysis from Monocular Videos
MEMFOF: High-Resolution Training for Memory-Efficient Multi-Frame Optical Flow Estimation
CycleVAR: Repurposing Autoregressive Model for Unsupervised One-Step Image Translation
WAFT: Warping-Alone Field Transforms for Optical Flow
CycleDistill: Bootstrapping Machine Translation using LLMs with Cyclical Distillation
Multimodal Fusion SLAM with Fourier Attention
토론
이 논문과 연결된 포럼 글이 아직 없습니다.