AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment
원문 초록
The speech-to-singing (STS) voice conversion task aims to generate singing samples corresponding to speech recordings while facing a major challenge: the alignment between the target (singing) pitch contour and the source (speech) content is difficult to learn in a text-free situation. This paper proposes AlignSTS, an STS model based on explicit cross-modal alignment, which views speech variance such as pitch and content as different modalities. Inspired by the mechanism of how humans will sing the lyrics to the melody, AlignSTS: 1) adopts a novel rhythm adaptor to predict the target rhythm representation to bridge the modality gap between content and pitch, where the rhythm representation is computed in a simple yet effective way and is quantized into a discrete space; and 2) uses the predicted rhythm representation to re-align the content based on cross-attention and conducts a cross-modal fusion for re-synthesize. Extensive experiments show that AlignSTS achieves superior performance in terms of both objective and subjective metrics. Audio samples are available at https://alignsts.github.io.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Unsupervised Rhythm and Voice Conversion to Improve ASR on Dysarthric Speech
DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learning
Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching
PromptEVC: Controllable Emotional Voice Conversion with Natural Language Prompts
Unsupervised Rhythm and Voice Conversion of Dysarthric to Healthy Speech for ASR
Skywork-R1V3 Technical Report
토론
이 논문과 연결된 포럼 글이 아직 없습니다.