End-to-end Learning for 3D Facial Animation from Raw Waveforms of Speech
원문 초록
We present a deep learning framework for real-time speech-driven 3D facial animation from just raw waveforms. Our deep neural network directly maps an input sequence of speech audio to a series of micro facial action unit activations and head rotations to drive a 3D blendshape face model. In particular, our deep model is able to learn the latent representations of time-varying contextual information and affective states within the speech. Hence, our model not only activates appropriate facial action units at inference to depict different utterance generating actions, in the form of lip movements, but also, without any assumption, automatically estimates emotional intensity of the speaker and reproduces her ever-changing affective states by adjusting strength of facial unit activations. For example, in a happy speech, the mouth opens wider than normal, while other facial units are relaxed; or in a surprised state, both eyebrows raise higher. Experiments on a diverse audiovisual corpus of different actors across a wide range of emotional states show interesting and promising results of our approach. Being speaker-independent, our generalized model is readily applicable to various tasks in human-machine interaction and animation.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
G2Face: High-Fidelity Reversible Face Anonymization via Generative and Geometric Priors
LeGO: Leveraging a Surface Deformation Network for Animatable Stylized Face Generation with One Example
ImFace++: A Sophisticated Nonlinear 3D Morphable Face Model with Implicit Neural Representations
GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians
A survey and classification of face alignment methods based on face models
Longitudinal Data and a Semantic Similarity Reward for Chest X-Ray Report Generation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.