DiffTalker: Co-driven audio-image diffusion for talking faces via intermediate landmarks
원문 초록
Generating realistic talking faces is a complex and widely discussed task with numerous applications. In this paper, we present DiffTalker, a novel model designed to generate lifelike talking faces through audio and landmark co-driving. DiffTalker addresses the challenges associated with directly applying diffusion models to audio control, which are traditionally trained on text-image pairs. DiffTalker consists of two agent networks: a transformer-based landmarks completion network for geometric accuracy and a diffusion-based face generation network for texture details. Landmarks play a pivotal role in establishing a seamless connection between the audio and image domains, facilitating the incorporation of knowledge from pre-trained diffusion models. This innovative approach efficiently produces articulate-speaking faces. Experimental results showcase DiffTalker's superior performance in producing clear and geometrically accurate talking faces, all without the need for additional alignment between audio and image features.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
FLUXSynID: A Framework for Identity-Controlled Synthetic Face Generation with Document and Live Images
MagicPortrait: Temporally Consistent Face Reenactment with 3D Geometric Guidance
ID-Booth: Identity-consistent Face Generation with Diffusion Models
FFaceNeRF: Few-shot Face Editing in Neural Radiance Fields
JoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing
DEGSTalk: Decomposed Per-Embedding Gaussian Fields for Hair-Preserving Talking Face Synthesis
토론
이 논문과 연결된 포럼 글이 아직 없습니다.