A Unified Framework for Multimodal, Multi-Part Human Motion Synthesis
원문 초록
The field has made significant progress in synthesizing realistic human motion driven by various modalities. Yet, the need for different methods to animate various body parts according to different control signals limits the scalability of these techniques in practical scenarios. In this paper, we introduce a cohesive and scalable approach that consolidates multimodal (text, music, speech) and multi-part (hand, torso) human motion generation. Our methodology unfolds in several steps: We begin by quantizing the motions of diverse body parts into separate codebooks tailored to their respective domains. Next, we harness the robust capabilities of pre-trained models to transcode multimodal signals into a shared latent space. We then translate these signals into discrete motion tokens by iteratively predicting subsequent tokens to form a complete sequence. Finally, we reconstruct the continuous actual motion from this tokenized sequence. Our method frames the multimodal motion generation challenge as a token prediction task, drawing from specialized codebooks based on the modality of the control signal. This approach is inherently scalable, allowing for the easy integration of new modalities. Extensive experiments demonstrated the effectiveness of our design, emphasizing its potential for broad application.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion Models
Motion Anything: Any to Motion Generation
Free-T2M: Frequency Enhanced Text-to-Motion Diffusion Model With Consistency Loss
KETA: Kinematic-Phrases-Enhanced Text-to-Motion Generation via Fine-grained Alignment
Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes
BAD: Bidirectional Auto-regressive Diffusion for Text-to-Motion Generation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.