Using Rater and System Metadata to Explain Variance in the VoiceMOS Challenge 2022 Dataset
원문 초록
Non-reference speech quality models are important for a growing number of applications. The VoiceMOS 2022 challenge provided a dataset of synthetic voice conversion and text-to-speech samples with subjective labels. This study looks at the amount of variance that can be explained in subjective ratings of speech quality from metadata and the distribution imbalances of the dataset. Speech quality models were constructed using wav2vec 2.0 with additional metadata features that included rater groups and system identifiers and obtained competitive metrics including a Spearman rank correlation coefficient (SRCC) of 0.934 and MSE of 0.088 at the system-level, and 0.877 and 0.198 at the utterance-level. Using data and metadata that the test restricted or blinded further improved the metrics. A metadata analysis showed that the system-level metrics do not represent the model's system-level prediction as a result of the wide variation in the number of utterances used for each system on the validation and test datasets. We conclude that, in general, conditions should have enough utterances in the test set to bound the sample mean error, and be relatively balanced in utterance count between systems, otherwise the utterance-level metrics may be more reliable and interpretable.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Metis: A Foundation Speech Generation Model with Masked Generative Pre-training
Overview of the Amphion Toolkit (v0.2)
SALF-MOS: Speaker Agnostic Latent Features Downsampled for MOS Prediction
An Exhaustive Evaluation of TTS- and VC-based Data Augmentation for ASR
ASVspoof 5: Design, Collection and Validation of Resources for Spoofing, Deepfake, and Adversarial Attack Detection Using Crowdsourced Speech
Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement
토론
이 논문과 연결된 포럼 글이 아직 없습니다.