Face-StyleSpeech: Enhancing Zero-shot Speech Synthesis from Face Images with Improved Face-to-Speech Mapping
원문 초록
Generating speech from a face image is crucial for developing virtual humans capable of interacting using their unique voices, without relying on pre-recorded human speech. In this paper, we propose Face-StyleSpeech, a zero-shot Text-To-Speech (TTS) synthesis model that generates natural speech conditioned on a face image rather than reference speech. We hypothesize that learning entire prosodic features from a face image poses a significant challenge. To address this, our TTS model incorporates both face and prosody encoders. The prosody encoder is specifically designed to model speech style characteristics that are not fully captured by the face image, allowing the face encoder to focus on extracting speaker-specific features such as timbre. Experimental results demonstrate that Face-StyleSpeech effectively generates more natural speech from a face image than baselines, even for unseen faces. Samples are available on our demo page.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
RapFlow-TTS: Rapid and High-Fidelity Text-to-Speech with Improved Consistency Flow Matching
InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems
ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching
Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models
MoonCast: High-Quality Zero-Shot Podcast Generation
Developing multilingual speech synthesis system for Ojibwe, Mi'kmaq, and Maliseet
토론
이 논문과 연결된 포럼 글이 아직 없습니다.