PyTorchKR Papers

최신 논문

총 5,401편
태그·기간 등 상세 필터

카테고리 탐색

로그인 후 이 태그 구독

arXiv:2606.30237v2cs.CL

네덜란드어 구음장애 연속 발화에 대한 인간과 자동 인식 비교: 사례 연구

Comparing Human and Automatic Recognition of Dutch Dysarthric Continuous Speech: A Case Study

개인화된 구음장애 음성 인식(DSR) 모델 개발을 목표로, 본 연구에서는 중증 구음장애를 가진 단일 화자의 네덜란드어 연속 낭독 발화 및 자발화에 대해 인간 청취자와 세 가지 최신 범용 자동 음성 인식(ASR) 시스템(Whisper-large-V3, Google Chirp 3, Omnilingual)의 인식 성능을 비교하였다. 결과에 따르면 인간 청취자와 세 가지 범용 ASR 시스템 모두 평균 단어 오류율(WER)이 70%를 초과하여, DSR이 인간과 ASR 시스템 모두에게 매우 어려운 과제임을 보여준다. 구음장애 발화에 대한 미세조정은 WER을 크게 감소시켰다. 전체 WER이 여전히 상당히 높지만(>23%), 개인화된 DSR 모델은 인간 청취자를 능가하였으며, 구음장애 화자의 일상 의사소통 지원에 활용 가능한 수준에 점차 가까워지고 있다. 향후 연구는 자발화에서의 개인화 DSR 성능 향상과 낭독 발화에서의 긴 발화 처리에 집중해야 하며, 특정 음소에 대한 심층적인 분석이 필요하다.

arXiv:2606.30356v2cs.CL

OLIVE: 음성 자기지도학습을 위한 뷰 증강 잠재 예측과 파형 재구성

OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL

우리는 분석과 합성 목적을 함께 최적화하는 자기지도 음성 표현 학습 프레임워크인 OLIVE(Online Latent prediction with Invariant Views and rEconstruction)를 제안한다. OLIVE는 단일 인코더 내의 기능적 분리를 통해 마스킹된 잠재 예측과 파형 복원을 결합한다. 복원은 인코더의 초기 특징이 신호 수준의 정보를 유지하도록 제약하고, 마스킹된 잠재 예측은 강건한 하위 작업 성능을 위해 이후의 문맥 표현을 형성하며, 파형 뷰 증강은 이들이 학습하는 불변성을 정의한다. 우리가 아는 한, OLIVE는 파형을 직접 복원하고, 사전학습 후에도 보코더를 유지하며, 판별 과제에서도 경쟁력을 갖는 최초의 음성 SSL 모델이다. 복원을 추가해도 마스킹된 잠재 예측만 사용할 때의 음성 인식 성능은 대체로 유지되면서, 파형 복원과 화자 관련 과제는 향상된다. 파형 mixup과 이득 증강을 추가하면 표현을 화자, 생성, 복원 과제 쪽으로 더 이동시키는 제어 가능한 방법을 제공하면서도 인식 및 의미 과제에서 경쟁력을 유지한다.

arXiv:2606.25990v2cs.CL

SpeechEQ: 사회적 인식을 갖춘 음성 대화 모델의 감성 지능 지수 벤치마킹

SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models

멀티모달 대화 시스템이 음성 상호작용에 점점 더 많이 참여함에 따라, 준언어적 사회 신호를 다루는 능력이 자연스러운 인간-AI 소통의 핵심 병목이 되었다. 그러나 기존의 기계 감성 지능 평가는 고립된 텍스트나 수동적인 음향 인식을 통해서만 추론을 평가하여, 능동적인 다중 턴 대화에 필요한 복잡한 교차 모달 추론을 간과한다. 우리는 음성-언어 모델(SLM)의 사회언어학적 추론을 평가하기 위해 설계된 포괄적 프레임워크인 \textsc{SpeechEQ}를 소개한다. 이 프레임워크는 EQ-i 2.0 이론에 기반한 15개 감성 지능(EQ) 하위 척도에 걸친 2,265개 대화로 구성된 검증된 데이터셋과, 인간 EQ 평가에서 착안하여 제안한 음성 EQ(SEQ) 점수로 측정되는 다중 턴 평가 프로토콜을 포함한다. 실험은 기존 음성 감정 인식과 종단간 음성-언어 모델 모두가 음성을 통한 준언어적 단서를 이해하고 적용하는 방식에 한계가 있음을 보여준다. 종단간 아키텍처가 캐스케이드 시스템보다 우수하지만, \textsc{SpeechEQ}는 현재의 멀티모달 모델이 텍스트에 의존하는 ``모달리티 지름길'', 정렬로 인해 유발되는 ``안전 함정'', 그리고 ``맥락 건망증''에 여전히 병목이 있음을 드러내며, 진정으로 감성을 인식하는 AI로 나아가는 데 장벽이 있음을 부각한다. 우리의 벤치마크는 https://huggingface.co/datasets/SpeechEQ/SpeechEQ 에서, 데모 페이지는 https://speecheq.github.io 에서 접근할 수 있다.

arXiv:2609.28060v2cs.CL

자기지도 음성 모델을 이용한 L2 발음 편차 측정용 원어민 기준 좌표 기하학

A Native-Reference Coordinate Geometry for L2 Pronunciation Deviation Using Self-Supervised Speech Models

Self-supervised speech models encode rich phonetic information, but it remains unclear how to transform this information into interpretable metrics for second-language (L2) pronunciation assessment in spontaneous speech. We propose a native-reference coordinate geometry in which phone-class averages from native speech define a low-dimensional reference subspace, and L2 speech is evaluated by its distance to matching native phone-class coordinates. Unlike prior distance-based approaches, our method does not require parallel recordings with matched linguistic content or dedicated pronunciation labels. Across different self-supervised encoders and modeling choices, the resulting native-reference distances show negative Spearman correlations up to -0.5 with speaking proficiency, indicating that higher-proficiency speakers tend to have smaller distances to the corresponding native phone-class coordinates.

arXiv:2607.06540v5cs.CL

계층적 음향-의미 모델링: 완전 이중 통화 음성 언어 모델을 위한 모달리티 분리와 의미 일관성

Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

완전 이중 통화(full-duplex) 음성 언어 모델(SLM)에서 음향·의미 모달리티가 깊은 파라미터 공간을 공유할 때 발생하는 그래디언트 충돌이 성능 저하의 근본 원인임을 밝히고, 이를 해결하기 위해 심층 레이어에서 두 모달리티를 분리하고 전용 의미 정렬 채널로 일관성을 유지하는 계층적 프레임워크 Lychee-FD를 제안한다. Spoken QA에서 +7.4%, FullDuplexBench 1.5에서 +28.5%의 성능 향상을 달성하며 추론 효율도 유지함을 입증했다.

arXiv:2411.14493v2cs.CL

From Statistical Methods to Pre-Trained Models; A Survey on Automatic Speech Recognition for Resource Scarce Urdu Language

Automatic Speech Recognition (ASR) technology has witnessed significant advancements in recent years, revolutionizing human-computer interactions. While major languages have benefited from these developments, lesser-resourced languages like Urdu face unique challenges. This paper provides an extensive exploration of the dynamic landscape of ASR research, focusing particularly on the resource-constrained Urdu language, which is widely spoken across South Asian nations. It outlines current research trends, technological advancements, and potential directions for future studies in Urdu ASR, aiming to pave the way for forthcoming researchers interested in this domain. By leveraging contemporary technologies, analyzing existing datasets, and evaluating effective algorithms and tools, the paper seeks to shed light on the unique challenges and opportunities associated with Urdu language processing and its integration into the broader field of speech research.

arXiv:2507.13155

NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech

Current expressive speech synthesis models are constrained by the limited availability of open-source datasets containing diverse nonverbal vocalizations (NVs). In this work, we introduce NonverbalTTS (NVTTS), a 17-hour open-access dataset annotated with 10 types of NVs (e.g., laughter, coughs) and 8 emotional categories. The dataset is derived from popular sources, VoxCeleb and Expresso, using automated detection followed by human validation. We propose a comprehensive pipeline that integrates automatic speech recognition (ASR), NV tagging, emotion classification, and a fusion algorithm to merge transcriptions from multiple annotators. Fine-tuning open-source text-to-speech (TTS) models on the NVTTS dataset achieves parity with closed-source systems such as CosyVoice2, as measured by both human evaluation and automatic metrics, including speaker similarity and NV fidelity. By releasing NVTTS and its accompanying annotation guidelines, we address a key bottleneck in expressive TTS research. The dataset is available at https://huggingface.co/datasets/deepvk/NonverbalTTS.

arXiv:2507.12701

Task-Specific Audio Coding for Machines: Machine-Learned Latent Features Are Codes for That Machine

Neural audio codecs, leveraging quantization algorithms, have significantly impacted various speech/audio tasks. While high-fidelity reconstruction is paramount for human perception, audio coding for machines (ACoM) prioritizes efficient compression and downstream task performance, disregarding perceptual nuances. This work introduces an efficient ACoM method that can compress and quantize any chosen intermediate feature representation of an already trained speech/audio downstream model. Our approach employs task-specific loss guidance alongside residual vector quantization (RVQ) losses, providing ultra-low bitrates (i.e., less than 200 bps) with a minimal loss of the downstream model performance. The resulting tokenizer is adaptable to various bitrates and model sizes for flexible deployment. Evaluated on automatic speech recognition and audio classification, our method demonstrates its efficacy and potential for broader task and architectural applicability through appropriate regularization.

arXiv:2507.10860

WhisperKit: On-device Real-time ASR with Billion-Scale Transformers

Real-time Automatic Speech Recognition (ASR) is a fundamental building block for many commercial applications of ML, including live captioning, dictation, meeting transcriptions, and medical scribes. Accuracy and latency are the most important factors when companies select a system to deploy. We present WhisperKit, an optimized on-device inference system for real-time ASR that significantly outperforms leading cloud-based systems. We benchmark against server-side systems that deploy a diverse set of models, including a frontier model (OpenAI gpt-4o-transcribe), a proprietary model (Deepgram nova-3), and an open-source model (Fireworks large-v3-turbo).Our results show that WhisperKit matches the lowest latency at 0.46s while achieving the highest accuracy 2.2% WER. The optimizations behind the WhisperKit system are described in detail in this paper.

arXiv:2507.08832

A Hybrid Machine Learning Framework for Optimizing Crop Selection via Agronomic and Economic Forecasting

Farmers in developing regions like Karnataka, India, face a dual challenge: navigating extreme market and climate volatility while being excluded from the digital revolution due to literacy barriers. This paper presents a novel decision support system that addresses both challenges through a unique synthesis of machine learning and human-computer interaction. We propose a hybrid recommendation engine that integrates two predictive models: a Random Forest classifier to assess agronomic suitability based on soil, climate, and real-time weather data, and a Long Short-Term Memory (LSTM) network to forecast market prices for agronomically viable crops. This integrated approach shifts the paradigm from "what can grow?" to "what is most profitable to grow?", providing a significant advantage in mitigating economic risk. The system is delivered through an end-to-end, voice-based interface in the local Kannada language, leveraging fine-tuned speech recognition and high-fidelity speech synthesis models to ensure accessibility for low-literacy users. Our results show that the Random Forest model achieves 98.5% accuracy in suitability prediction, while the LSTM model forecasts harvest-time prices with a low margin of error. By providing data-driven, economically optimized recommendations through an inclusive interface, this work offers a scalable and impactful solution to enhance the financial resilience of marginalized farming communities.

arXiv:2507.06060

VisualSpeaker: Visually-Guided 3D Avatar Lip Synthesis

Realistic, high-fidelity 3D facial animations are crucial for expressive avatar systems in human-computer interaction and accessibility. Although prior methods show promising quality, their reliance on the mesh domain limits their ability to fully leverage the rapid visual innovations seen in 2D computer vision and graphics. We propose VisualSpeaker, a novel method that bridges this gap using photorealistic differentiable rendering, supervised by visual speech recognition, for improved 3D facial animation. Our contribution is a perceptual lip-reading loss, derived by passing photorealistic 3D Gaussian Splatting avatar renders through a pre-trained Visual Automatic Speech Recognition model during training. Evaluation on the MEAD dataset demonstrates that VisualSpeaker improves both the standard Lip Vertex Error metric by 56.1% and the perceptual quality of the generated animations, while retaining the controllability of mesh-driven animation. This perceptual focus naturally supports accurate mouthings, essential cues that disambiguate similar manual signs in sign language avatars.

arXiv:2507.01765공식 코드 · Papers with Code

First Steps Towards Voice Anonymization for Code-Switching Speech

The goal of voice anonymization is to modify an audio such that the true identity of its speaker is hidden. Research on this task is typically limited to the same English read speech datasets, thus the efficacy of current methods for other types of speech data remains unknown. In this paper, we present the first investigation of voice anonymization for the multilingual phenomenon of code-switching speech. We prepare two corpora for this task and propose adaptations to a multilingual anonymization model to make it applicable for code-switching speech. By testing the anonymization performance of this and two language-independent methods on the datasets, we find that only the multilingual system performs well in terms of privacy and utility preservation. Furthermore, we observe challenges in performing utility evaluations on this data because of its spontaneous character and the limited code-switching support by the multilingual speech recognition model.

arXiv:2507.00966공식 코드 · Papers with Code

MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancement

With the advent of new sequence models like Mamba and xLSTM, several studies have shown that these models match or outperform state-of-the-art models in single-channel speech enhancement, automatic speech recognition, and self-supervised audio representation learning. However, prior research has demonstrated that sequence models like LSTM and Mamba tend to overfit to the training set. To address this issue, previous works have shown that adding self-attention to LSTMs substantially improves generalization performance for single-channel speech enhancement. Nevertheless, neither the concept of hybrid Mamba and time-frequency attention models nor their generalization performance have been explored for speech enhancement. In this paper, we propose a novel hybrid architecture, MambAttention, which combines Mamba and shared time- and frequency-multi-head attention modules for generalizable single-channel speech enhancement. To train our model, we introduce VoiceBank+Demand Extended (VB-DemandEx), a dataset inspired by VoiceBank+Demand but with more challenging noise types and lower signal-to-noise ratios. Trained on VB-DemandEx, our proposed MambAttention model significantly outperforms existing state-of-the-art LSTM-, xLSTM-, Mamba-, and Conformer-based systems of similar complexity across all reported metrics on two out-of-domain datasets: DNS 2020 and EARS-WHAM_v2, while matching their performance on the in-domain dataset VB-DemandEx. Ablation studies highlight the role of weight sharing between the time- and frequency-multi-head attention modules for generalization performance. Finally, we explore integrating the shared time- and frequency-multi-head attention modules with LSTM and xLSTM, which yields a notable performance improvement on the out-of-domain datasets. However, our MambAttention model remains superior on both out-of-domain datasets across all reported evaluation metrics.

arXiv:2506.14973

Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition

Recent studies have demonstrated that prompting large language models (LLM) with audio encodings enables effective speech recognition capabilities. However, the ability of Speech LLMs to comprehend and process multi-channel audio with spatial cues remains a relatively uninvestigated area of research. In this work, we present directional-SpeechLlama, a novel approach that leverages the microphone array of smart glasses to achieve directional speech recognition, source localization, and bystander cross-talk suppression. To enhance the model's ability to understand directivity, we propose two key techniques: serialized directional output training (S-DOT) and contrastive direction data augmentation (CDDA). Experimental results show that our proposed directional-SpeechLlama effectively captures the relationship between textual cues and spatial audio, yielding strong performance in both speech recognition and source localization tasks.

arXiv:2506.13339

NTU Speechlab LLM-Based Multilingual ASR System for Interspeech MLC-SLM Challenge 2025

This report details the NTU Speechlab system developed for the Interspeech 2025 Multilingual Conversational Speech and Language Model (MLC-SLM) Challenge (Task I), where we achieved 5th place. We present comprehensive analyses of our multilingual automatic speech recognition system, highlighting key advancements in model architecture, data selection, and training strategies. In particular, language-specific prompts and model averaging techniques were instrumental in boosting system performance across diverse languages. Compared to the initial baseline system, our final model reduced the average Mix Error Rate from 20.2% to 10.6%, representing an absolute improvement of 9.6% (a relative improvement of 48%) on the evaluation set. Our results demonstrate the effectiveness of our approach and offer practical insights for future Speech Large Language Models.

arXiv:2506.11630

Lightweight and Robust Multi-Channel End-to-End Speech Recognition with Spherical Harmonic Transform

This paper presents SHTNet, a lightweight spherical harmonic transform (SHT) based framework, which is designed to address cross-array generalization challenges in multi-channel automatic speech recognition (ASR) through three key innovations. First, SHT based spatial sound field decomposition converts microphone signals into geometry-invariant spherical harmonic coefficients, isolating signal processing from array geometry. Second, the Spatio-Spectral Attention Fusion Network (SSAFN) combines coordinate-aware spatial modeling, refined self-attention channel combinator, and spectral noise suppression without conventional beamforming. Third, Rand-SHT training enhances robustness through random channel selection and array geometry reconstruction. The system achieves 39.26\% average CER across heterogeneous arrays (e.g., circular, square, and binaural) on datasets including Aishell-4, Alimeeting, and XMOS, with 97.1\% fewer computations than conventional neural beamformers.

arXiv:2506.11091

Customizing Speech Recognition Model with Large Language Model Feedback

Automatic speech recognition (ASR) systems have achieved strong performance on general transcription tasks. However, they continue to struggle with recognizing rare named entities and adapting to domain mismatches. In contrast, large language models (LLMs), trained on massive internet-scale datasets, are often more effective across a wide range of domains. In this work, we propose a reinforcement learning based approach for unsupervised domain adaptation, leveraging unlabeled data to enhance transcription quality, particularly the named entities affected by domain mismatch, through feedback from a LLM. Given contextual information, our framework employs a LLM as the reward model to score the hypotheses from the ASR model. These scores serve as reward signals to fine-tune the ASR model via reinforcement learning. Our method achieves a 21\% improvement on entity word error rate over conventional self-training methods.

arXiv:2506.11089

Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLM

Automatic speech recognition (ASR) models rely on high-quality transcribed data for effective training. Generating pseudo-labels for large unlabeled audio datasets often relies on complex pipelines that combine multiple ASR outputs through multi-stage processing, leading to error propagation, information loss and disjoint optimization. We propose a unified multi-ASR prompt-driven framework using postprocessing by either textual or speech-based large language models (LLMs), replacing voting or other arbitration logic for reconciling the ensemble outputs. We perform a comparative study of multiple architectures with and without LLMs, showing significant improvements in transcription accuracy compared to traditional methods. Furthermore, we use the pseudo-labels generated by the various approaches to train semi-supervised ASR models for different datasets, again showing improved performance with textual and speechLLM transcriptions compared to baselines.

arXiv:2506.11079

Improving Child Speech Recognition and Reading Mistake Detection by Using Prompts

Automatic reading aloud evaluation can provide valuable support to teachers by enabling more efficient scoring of reading exercises. However, research on reading evaluation systems and applications remains limited. We present a novel multimodal approach that leverages audio and knowledge from text resources. In particular, we explored the potential of using Whisper and instruction-tuned large language models (LLMs) with prompts to improve transcriptions for child speech recognition, as well as their effectiveness in downstream reading mistake detection. Our results demonstrate the effectiveness of prompting Whisper and prompting LLM, compared to the baseline Whisper model without prompting. The best performing system achieved state-of-the-art recognition performance in Dutch child read speech, with a word error rate (WER) of 5.1%, improving the baseline WER of 9.4%. Furthermore, it significantly improved reading mistake detection, increasing the F1 score from 0.39 to 0.73.

arXiv:2506.20288

Lightweight Target-Speaker-Based Overlap Transcription for Practical Streaming ASR

Overlapping speech remains a major challenge for automatic speech recognition (ASR) in real-world applications, particularly in broadcast media with dynamic, multi-speaker interactions. We propose a light-weight, target-speaker-based extension to an existing streaming ASR system to enable practical transcription of overlapping speech with minimal computational overhead. Our approach combines a speaker-independent (SI) model for standard operation with a speaker-conditioned (SC) model selectively applied in overlapping scenarios. Overlap detection is achieved using a compact binary classifier trained on frozen SI model output, offering accurate segmentation at negligible cost. The SC model employs Feature-wise Linear Modulation (FiLM) to incorporate speaker embeddings and is trained on synthetically mixed data to transcribe only the target speaker. Our method supports dynamic speaker tracking and reuses existing modules with minimal modifications. Evaluated on a challenging set of Czech television debates with 16% overlap, the system reduced WER on overlapping segments from 68.0% (baseline) to 35.78% while increasing total computational load by only 44%. The proposed system offers an effective and scalable solution for overlap transcription in continuous ASR services.