Improving Speech Related Facial Action Unit Recognition by Audiovisual Information Fusion
원문 초록
It is challenging to recognize facial action unit (AU) from spontaneous facial displays, especially when they are accompanied by speech. The major reason is that the information is extracted from a single source, i.e., the visual channel, in the current practice. However, facial activity is highly correlated with voice in natural human communications. Instead of solely improving visual observations, this paper presents a novel audiovisual fusion framework, which makes the best use of visual and acoustic cues in recognizing speech-related facial AUs. In particular, a dynamic Bayesian network (DBN) is employed to explicitly model the semantic and dynamic physiological relationships between AUs and phonemes as well as measurement uncertainty. A pilot audiovisual AU-coded database has been collected to evaluate the proposed framework, which consists of a "clean" subset containing frontal faces under well controlled circumstances and a challenging subset with large head movements and occlusions. Experiments on this database have demonstrated that the proposed framework yields significant improvement in recognizing speech-related AUs compared to the state-of-the-art visual-based methods especially for those AUs whose visual observations are impaired during speech, and more importantly also outperforms feature-level fusion methods by explicitly modeling and exploiting physiological relationships between AUs and phonemes.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample
Norface: Improving Facial Expression Analysis by Identity Normalization
Representation Learning and Identity Adversarial Training for Facial Behavior Understanding
Multi-scale Dynamic and Hierarchical Relationship Modeling for Facial Action Units Recognition
GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing
AUFormer: Vision Transformers are Parameter-Efficient Facial Action Unit Detectors
토론
이 논문과 연결된 포럼 글이 아직 없습니다.