It Takes Two: Real-time Co-Speech Two-person's Interaction Generation via Reactive Auto-regressive Diffusion Model
원문 초록
Conversational scenarios are very common in real-world settings, yet existing co-speech motion synthesis approaches often fall short in these contexts, where one person's audio and gestures will influence the other's responses. Additionally, most existing methods rely on offline sequence-to-sequence frameworks, which are unsuitable for online applications. In this work, we introduce an audio-driven, auto-regressive system designed to synthesize dynamic movements for two characters during a conversation. At the core of our approach is a diffusion-based full-body motion synthesis model, which is conditioned on the past states of both characters, speech audio, and a task-oriented motion trajectory input, allowing for flexible spatial control. To enhance the model's ability to learn diverse interactions, we have enriched existing two-person conversational motion datasets with more dynamic and interactive motions. We evaluate our system through multiple experiments to show it outperforms across a variety of tasks, including single and two-person co-speech motion generation, as well as interactive motion generation. To the best of our knowledge, this is the first system capable of generating interactive full-body motions for two characters from speech in an online manner.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion Models
Motion Anything: Any to Motion Generation
Free-T2M: Frequency Enhanced Text-to-Motion Diffusion Model With Consistency Loss
KETA: Kinematic-Phrases-Enhanced Text-to-Motion Generation via Fine-grained Alignment
Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes
BAD: Bidirectional Auto-regressive Diffusion for Text-to-Motion Generation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.