Synthesising 3D Facial Motion from "In-the-Wild" Speech
원문 초록
Synthesising 3D facial motion from speech is a crucial problem manifesting in a multitude of applications such as computer games and movies. Recently proposed methods tackle this problem in controlled conditions of speech. In this paper, we introduce the first methodology for 3D facial motion synthesis from speech captured in arbitrary recording conditions ("in-the-wild") and independent of the speaker. For our purposes, we captured 4D sequences of people uttering 500 words, contained in the Lip Reading Words (LRW) a publicly available large-scale in-the-wild dataset, and built a set of 3D blendshapes appropriate for speech. We correlate the 3D shape parameters of the speech blendshapes to the LRW audio samples by means of a novel time-warping technique, named Deep Canonical Attentional Warping (DCAW), that can simultaneously learn hierarchical non-linear representations and a warping path in an end-to-end manner. We thoroughly evaluate our proposed methods, and show the ability of a deep learning model to synthesise 3D facial motion in handling different speakers and continuous speech signals in uncontrolled conditions.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
DeepGesture: A conversational gesture synthesis system based on emotions and semantics
VolumetricSMPL: A Neural Volumetric Body Model for Efficient Interactions, Contacts, and Collisions
DuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modeling
Transforming faces into video stories -- VideoFace2.0
Generative AI for Character Animation: A Comprehensive Survey of Techniques, Applications, and Future Directions
Reinforcement learning-based motion imitation for physiologically plausible musculoskeletal motor control
토론
이 논문과 연결된 포럼 글이 아직 없습니다.