A novel multimodal dynamic fusion network for disfluency detection in spoken utterances
원문 초록
Disfluency, though originating from human spoken utterances, is primarily studied as a uni-modal text-based Natural Language Processing (NLP) task. Based on early-fusion and self-attention-based multimodal interaction between text and acoustic modalities, in this paper, we propose a novel multimodal architecture for disfluency detection from individual utterances. Our architecture leverages a multimodal dynamic fusion network that adds minimal parameters over an existing text encoder commonly used in prior art to leverage the prosodic and acoustic cues hidden in speech. Through experiments, we show that our proposed model achieves state-of-the-art results on the widely used English Switchboard for disfluency detection and outperforms prior unimodal and multimodal systems in literature by a significant margin. In addition, we make a thorough qualitative analysis and show that, unlike text-only systems, which suffer from spurious correlations in the data, our system overcomes this problem through additional cues from speech signals. We make all our codes publicly available on GitHub.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding
I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interaction
OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer
Towards Explainable Multimodal Depression Recognition for Clinical Interviews
토론
이 논문과 연결된 포럼 글이 아직 없습니다.