Compositional 3D-aware Video Generation with LLM Director
원문 초록
Significant progress has been made in text-to-video generation through the use of powerful generative models and large-scale internet data. However, substantial challenges remain in precisely controlling individual concepts within the generated video, such as the motion and appearance of specific characters and the movement of viewpoints. In this work, we propose a novel paradigm that generates each concept in 3D representation separately and then composes them with priors from Large Language Models (LLM) and 2D diffusion models. Specifically, given an input textual prompt, our scheme consists of three stages: 1) We leverage LLM as the director to first decompose the complex query into several sub-prompts that indicate individual concepts within the video~(\textit{e.g.}, scene, objects, motions), then we let LLM to invoke pre-trained expert models to obtain corresponding 3D representations of concepts. 2) To compose these representations, we prompt multi-modal LLM to produce coarse guidance on the scales and coordinates of trajectories for the objects. 3) To make the generated frames adhere to natural image distribution, we further leverage 2D diffusion priors and use Score Distillation Sampling to refine the composition. Extensive experiments demonstrate that our method can generate high-fidelity videos from text with diverse motion and flexible control over each concept. Project page: \url{https://aka.ms/c3v}.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation
InfLVG: Reinforce Inference-Time Consistent Long Video Generation with GRPO
LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation
On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devices
VPO: Aligning Text-to-Video Generation Models with Prompt Optimization
RecTable: Fast Modeling Tabular Data with Rectified Flow
토론
이 논문과 연결된 포럼 글이 아직 없습니다.