Visual Speech Recognition Using PCA Networks and LSTMs in a Tandem GMM-HMM System
원문 초록
Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches - extracted from the video data - to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79% and 73%, respectively, as compared to the baseline of 74% on cross-validation.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
BeaverTalk: Oregon State University's IWSLT 2025 Simultaneous Speech Translation System
Transfer Learning from Visual Speech Recognition to Mouthing Recognition in German Sign Language
VisualSpeaker: Visually-Guided 3D Avatar Lip Synthesis
ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition
Cocktail-Party Audio-Visual Speech Recognition
CNVSRC 2024: The Second Chinese Continuous Visual Speech Recognition Challenge
토론
이 논문과 연결된 포럼 글이 아직 없습니다.