Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning
원문 초록
Temporal observations such as videos contain essential information about the dynamics of the underlying scene, but they are often interleaved with inessential, predictable details. One way of dealing with this problem is by focusing on the most informative moments in a sequence. We propose a model that learns to discover these important events and the times when they occur and uses them to represent the full sequence. We do so using a hierarchical Keyframe-Inpainter (KeyIn) model that first generates a video's keyframes and then inpaints the rest by generating the frames at the intervening times. We propose a fully differentiable formulation to efficiently learn this procedure. We show that KeyIn finds informative keyframes in several datasets with different dynamics and visual properties. KeyIn outperforms other recent hierarchical predictive models for planning. For more details, please see the project website at \url{https://sites.google.com/view/keyin}.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Epona: Autoregressive Diffusion World Model for Autonomous Driving
Autoregression-free video prediction using diffusion model for mitigating error propagation
Programmatic Video Prediction Using Large Language Models
SRVP: Strong Recollection Video Prediction Model Using Attention-Based Spatiotemporal Correlation Fusion
SyncVP: Joint Diffusion for Synchronous Multi-Modal Video Prediction
Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Models
토론
이 논문과 연결된 포럼 글이 아직 없습니다.