Recently, the recognition task of spontaneous facial micro-expressions has attracted much attention with its various real-world applications. Plenty of handcrafted or learned features have been employed for a variety of classifiers and achieved promising performances for recognizing micro-expressions. However, the micro-expression recognition is still challenging due to the subtle spatiotemporal changes of micro-expressions. To exploit the merits of deep learning, we propose a novel deep recurrent convolutional networks based micro-expression recognition approach, capturing the spatial-temporal deformations of micro-expression sequence. Specifically, the proposed deep model is constituted of several recurrent convolutional layers for extracting visual features and a classificatory layer for recognition. It is optimized by an end-to-end manner and obviates manual feature design. To handle sequential data, we exploit two types of extending the connectivity of convolutional networks across temporal domain, in which the spatiotemporal deformations are modeled in views of facial appearance and geometry separately. Besides, to overcome the shortcomings of limited and imbalanced training samples, temporal data augmentation strategies as well as a balanced loss are jointly used for our deep network. By performing the experiments on three spontaneous micro-expression datasets, we verify the effectiveness of our proposed micro-expression recognition approach compared to the state-of-the-art methods.
Cross-database micro-expression recognition (CDMER) is one of recently emerging and interesting problem in micro-expression analysis. CDMER is more challenging than the conventional micro-expression recognition (MER), because the training and testing samples in CDMER come from different micro-expression databases, resulting in the inconsistency of the feature distributions between the training and testing sets. In this paper, we contribute to this topic from three aspects. First, we establish a CDMER experimental evaluation protocol aiming to allow the researchers to conveniently work on this topic and provide a standard platform for evaluating their proposed methods. Second, we conduct benchmark experiments by using NINE state-of-the-art domain adaptation (DA) methods and SIX popular spatiotemporal descriptors for respectively investigating CDMER problem from two different perspectives. Third, we propose a novel DA method called region selective transfer regression (RSTR) to deal with the CDMER task. Our RSTR takes advantage of one important cue for recognizing micro-expressions, i.e., the different contributions of the facial local regions in MER. The overall superior performance of RSTR demonstrates that taking into consideration the important cues benefiting MER, e.g., the facial local region information, contributes to develop effective DA methods for dealing with CDMER problem.
Micro-expression, for its high objectivity in emotion detection, has emerged to be a promising modality in affective computing. Recently, deep learning methods have been successfully introduced into the micro-expression recognition area. Whilst the higher recognition accuracy achieved, substantial challenges in micro-expression recognition remain. The existence of micro expression in small-local areas on face and limited size of available databases still constrain the recognition accuracy on such emotional facial behavior. In this work, to tackle such challenges, we propose a novel attention mechanism called micro-attention cooperating with residual network. Micro-attention enables the network to learn to focus on facial areas of interest covering different action units. Moreover, coping with small datasets, the micro-attention is designed without adding noticeable parameters while a simple yet efficient transfer learning approach is together utilized to alleviate the overfitting risk. With extensive experimental evaluations on three benchmarks (CASMEII, SAMM and SMIC) and post-hoc feature visualizations, we demonstrate the effectiveness of the proposed micro-attention and push the boundary of automatic recognition of micro-expression.
Recently, there are increasing interests in inferring mirco-expression from facial image sequences. Due to subtle facial movement of micro-expressions, feature extraction has become an important and critical issue for spontaneous facial micro-expression recognition. Recent works usually used spatiotemporal local binary pattern for micro-expression analysis. However, the commonly used spatiotemporal local binary pattern considers dynamic texture information to represent face images while misses the shape attribute of face images. On the other hand, their works extracted the spatiotemporal features from the global face regions, which ignore the discriminative information between two micro-expression classes. The above-mentioned problems seriously limit the application of spatiotemporal local binary pattern on micro-expression recognition. In this paper, we propose a discriminative spatiotemporal local binary pattern based on an improved integral projection to resolve the problems of spatiotemporal local binary pattern for micro-expression recognition. Firstly, we develop an improved integral projection for preserving the shape attribute of micro-expressions. Furthermore, an improved integral projection is incorporated with local binary pattern operators across spatial and temporal domains. Specifically, we extract the novel spatiotemporal features incorporating shape attributes into spatiotemporal texture features. For increasing the discrimination of micro-expressions, we propose a new feature selection based on Laplacian method to extract the discriminative information for facial micro-expression recognition. Intensive experiments are conducted on three availably published micro-expression databases. We compare our method with the state-of-the-art algorithms. Experimental results demonstrate that our proposed method achieves promising performance for micro-expression recognition.
In this paper, we investigate the cross-database micro-expression recognition problem, where the training and testing samples are from two different micro-expression databases. Under this setting, the training and testing samples would have different feature distributions and hence the performance of most existing micro-expression recognition methods may decrease greatly. To solve this problem, we propose a simple yet effective method called Target Sample Re-Generator (TSRG) in this paper. By using TSRG, we are able to re-generate the samples from target micro-expression database and the re-generated target samples would share same or similar feature distributions with the original source samples. For this reason, we can then use the classifier learned based on the labeled source samples to accurately predict the micro-expression categories of the unlabeled target samples. To evaluate the performance of the proposed TSRG method, extensive cross-database micro-expression recognition experiments designed based on SMIC and CASME II databases are conducted. Compared with recent state-of-the-art cross-database emotion recognition methods, the proposed TSRG achieves more promising results.
Micro-expressions are brief spontaneous facial expressions that appear on a face when a person conceals an emotion, making them different to normal facial expressions in subtlety and duration. Currently, emotion classes within the CASME II dataset are based on Action Units and self-reports, creating conflicts during machine learning training. We will show that classifying expressions using Action Units, instead of predicted emotion, removes the potential bias of human reporting. The proposed classes are tested using LBP-TOP, HOOF and HOG 3D feature descriptors. The experiments are evaluated on two benchmark FACS coded datasets: CASME II and SAMM. The best result achieves 86.35\% accuracy when classifying the proposed 5 classes on CASME II using HOG 3D, outperforming the result of the state-of-the-art 5-class emotional-based classification in CASME II. Results indicate that classification based on Action Units provides an objective method to improve micro-expression recognition.
Despite recent interest and advances in facial micro-expression research, there is still plenty room for improvement in terms of micro-expression recognition. Conventional feature extraction approaches for micro-expression video consider either the whole video sequence or a part of it, for representation. However, with the high-speed video capture of micro-expressions (100-200 fps), are all frames necessary to provide a sufficiently meaningful representation? Is the luxury of data a bane to accurate recognition? A novel proposition is presented in this paper, whereby we utilize only two images per video: the apex frame and the onset frame. The apex frame of a video contains the highest intensity of expression changes among all frames, while the onset is the perfect choice of a reference frame with neutral expression. A new feature extractor, Bi-Weighted Oriented Optical Flow (Bi-WOOF) is proposed to encode essential expressiveness of the apex frame. We evaluated the proposed method on five micro-expression databases: CAS(ME)$^2$, CASME II, SMIC-HS, SMIC-NIR and SMIC-VIS. Our experiments lend credence to our hypothesis, with our proposed technique achieving a state-of-the-art F1-score recognition performance of 61% and 62% in the high frame rate CASME II and SMIC-HS databases respectively.
In this paper, we develop a new method that recognizes facial expressions, on the basis of an innovative local motion patterns feature, with three main contributions. The first one is the analysis of the face skin temporal elasticity and face deformations during expression. The second one is a unified approach for both macro and micro expression recognition. And, the third one is the step forward towards in-the-wild expression recognition, dealing with challenges such as various intensity and various expression activation patterns, illumination variation and small head pose variations. Our method outperforms state-of-the-art methods for micro expression recognition and positions itself among top-rank state-of-the-art methods for macro expression recognition.
When a person attempts to conceal an emotion, the genuine emotion is manifest as a micro-expression. Exploration of automatic facial micro-expression recognition systems is relatively new in the computer vision domain. This is due to the difficulty in implementing optimal feature extraction methods to cope with the subtlety and brief motion characteristics of the expression. Most of the existing approaches extract the subtle facial movements based on hand-crafted features. In this paper, we address the micro-expression recognition task with a convolutional neural network (CNN) architecture, which well integrates the features extracted from each video. A new feature descriptor, Optical Flow Features from Apex frame Network (OFF-ApexNet) is introduced. This feature descriptor combines the optical ow guided context with the CNN. Firstly, we obtain the location of the apex frame from each video sequence as it portrays the highest intensity of facial motion among all frames. Then, the optical ow information are attained from the apex frame and a reference frame (i.e., onset frame). Finally, the optical flow features are fed into a pre-designed CNN model for further feature enhancement as well as to carry out the expression classification. To evaluate the effectiveness of OFF-ApexNet, comprehensive evaluations are conducted on three public spontaneous micro-expression datasets (i.e., SMIC, CASME II and SAMM). The promising recognition result suggests that the proposed method can optimally describe the significant micro-expression details. In particular, we report that, in a multi-database with leave-one-subject-out cross-validation experimental protocol, the recognition performance reaches 74.60% of recognition accuracy and F-measure of 71.04%. We also note that this is the first work that performs cross-dataset validation on three databases in this domain.
Facial micro-expression (ME) recognition has posed a huge challenge to researchers for its subtlety in motion and limited databases. Recently, handcrafted techniques have achieved superior performance in micro-expression recognition but at the cost of domain specificity and cumbersome parametric tunings. In this paper, we propose an Enriched Long-term Recurrent Convolutional Network (ELRCN) that first encodes each micro-expression frame into a feature vector through CNN module(s), then predicts the micro-expression by passing the feature vector through a Long Short-term Memory (LSTM) module. The framework contains two different network variants: (1) Channel-wise stacking of input data for spatial enrichment, (2) Feature-wise stacking of features for temporal enrichment. We demonstrate that the proposed approach is able to achieve reasonably good performance, without data augmentation. In addition, we also present ablation studies conducted on the framework and visualizations of what CNN "sees" when predicting the micro-expression classes.
Micro-expressions are subtle facial movements that occur spontaneously when people try to conceal real emotions. Micro-expression recognition is crucial in many fields, including criminal analysis and psychotherapy. However, micro-expression recognition is challenging since micro-expressions have low intensity and public datasets are small in size. To this end, a three-stream temporal-shift attention network based on self-knowledge distillation is proposed in this paper. Firstly, to address the low intensity of muscle movements, we utilize learning-based motion magnification modules to enhance the intensity of muscle movements. Secondly, we employ efficient channel attention modules in the local-spatial stream to make the network focus on facial regions that are highly relevant to micro-expressions. In addition, temporal shift modules are used in the dynamic-temporal stream, which enables temporal modeling with no additional parameters by mixing motion information from two different temporal domains. Furthermore, we introduce self-knowledge distillation into the micro-expression recognition task by introducing auxiliary classifiers and using the deepest section of the network for supervision, encouraging all blocks to fully explore the features of the training set. Finally, extensive experiments are conducted on five publicly available micro-expression datasets. The experimental results demonstrate that our network outperforms other existing methods and achieves new state-of-the-art performance. Our code is available at https://github.com/GuanghaoZhu663/SKD-TSTSAN.
거시표정(macro-expression)과 달리, 미세표정(micro-expression)은 감정과 Action Unit(AU) 사이에 엄격하게 일관된 매핑 규칙을 따르지 않는다. 그 결과, 일부 미세표정은 동일한 AU를 공유하면서도 완전히 반대되는 감정 범주를 나타내어 시각적으로 매우 유사하다. 기존 미세표정 인식(MER) 방법은 대부분 명시적 얼굴 운동 단서(예: optical flow, 프레임 차이, AU 특징)에 의존하면서 암묵적 감정 정보는 무시한다. 이 문제를 해결하기 위해 본 논문은 MER를 위한 Motion Emotion Feature Decoupling Network(MEDN)를 제안한다. 우리는 운동 특징과 감정 특징을 각각 추출하는 이중 분기 프레임워크를 설계한다. 운동 분기에서는 AU 검출 태스크가 특징을 명시적 운동 영역으로 제한하고, 직교 손실(orthogonal loss)을 채택하여 운동-감정 특징 결합을 줄인다. 암묵적 감정 모델링을 위해, 다중 스케일 희소성 비율로 공간 토큰을 희소화하여 국소적 시간적 변화를 강조하는 Sparse Emotion Vision Transformer(SEVit)를 제안한다. 또한 분리된 운동 특징과 감정 특징을 적응적으로 융합하는 Collaborative Fusion Module(CoFM)을 개발한다. 세 개의 벤치마크 데이터셋에 대한 광범위한 실험을 통해 MEDN이 운동 특징과 감정 특징을 효과적으로 분리하고 우수한 인식 성능을 달성함을 검증하며, 인식 정확도와 일반화를 향상시키는 새로운 관점을 제시한다.
미세표정 인식은 기존 데이터셋의 소규모, 좁은 인구통계학적 범위, 제한된 감정 레이블로 인해 한계가 있다. 우리는 AU 기반 이미지-투-비디오 생성으로 구축된 합성 미세표정 데이터셋인 EquiME를 소개한다. EquiME는 다섯 가지 목표 감정에 걸쳐 15K개의 소스 얼굴 이미지로부터 생성된 75K개의 비디오를 포함하며, 자동으로 추론된 인구통계학적 메타데이터와 비디오 품질 측정값도 함께 제공된다. 우리는 프레임 쌍 유사도, 공간 변화, 무참조 지각 품질 메트릭과 함께 SAMM 및 CASME II에 대한 교차 데이터셋 미세표정 인식(MER) 실험을 통해 EquiME를 평가한다. EquiME로 학습된 모델은 SAMM 및 CASME II에서 경쟁력 있는 교차 데이터셋 성능을 달성하며, 평가된 네 가지 아키텍처 전반에 걸쳐 비교적 낮은 변동성을 보인다. 본 논문은 데이터셋 설계, 비디오 생성에 사용된 구조화된 AU 조건화 파이프라인, 그리고 EquiME를 합성 MER 자원으로 평가하는 데 필요한 실증적 근거에 초점을 맞춘다. 프로젝트 페이지: https://kirito-blade.github.io/me-vlm/
미세표정 인식은 미묘하고 짧게 지속되는 안면 근육 움직임 때문에 어렵다. 기존 방법은 apex-onset 프레임에 크게 의존하고, 세밀한 프레임 간 역학을 간과하며, 공간 정보와 시간 정보를 별도로 모델링하여 데이터셋 간 일반화를 제한한다. 이러한 과제를 해결하기 위해, 우리는 움직임 흐름과 적응적 안면 연결성을 공동으로 모델링하는 동적 ROI-AU 결합 시공간 네트워크인 STAG를 제안한다. 해당 프레임워크는 크기 기반 선택과 시간 어텐션을 사용하여 판별적 프레임에서 광학 흐름(optical flow)을 추출한다. 이중 분기 아키텍처는 구조화된 공간 추론을 위한 향상된 그래프 어텐션 네트워크와 시간 모델링을 위한 트랜스포머 인코더를 결합한다. 양방향 교차 어텐션 모듈은 공간 특징과 시간 특징의 상호 정제를 가능하게 하며, AU 유도 동적 연결성은 근육 활성화 패턴에 따라 안면 영역 상호작용을 적응한다. 트랜스포머는 apex 기반 접근법을 넘어 미묘한 시간 역학을 포착하여, 설명 가능한 미세표정 인식을 위한 의미적 일관성과 해석 가능성을 향상시킨다. 융합된 표현은 focal 손실을 사용하여 최적화되며, CASME II, 4DME, DFME, NaME, SAMM, SMIC-HS에서 평가된다. 광범위한 실험은 향상된 견고성, 일반화, 해석 가능성, 계산 효율성을 보여주며, 정확한 교차 데이터셋 미세표정 인식을 위한 적응적 관계 추론, AU 유도 동적 연결성, 깊은 시공간 특징 융합의 효과를 확인한다.
현실적인 시나리오에서의 미세표정 인식(MER)은 높은 시간적 민감도와 생태학적 타당성을 요구하지만, 기존 벤치마크는 대부분 실험실 통제 환경과 경직된 하드웨어 결합 센싱에 제한되어 있다. 우리는 자발적인 감정 누출을 유발하기 위해 다단계 스트레스 유도 프레임워크(인지적 및 사회적) 하에서 수집된 기기 간 Event-RGB 데이터셋인 CDER-SME를 제안한다. 독립적이고 분리된 센서를 이용한 재현 가능한 수집을 가능하게 하기 위해, 시간적 동기화와 랜드마크 기반 공간 정합을 위한 하드웨어에 구애받지 않는 정렬 파이프라인을 제공한다. CDER-SME는 92명의 피험자와 1,963개의 전문가 주석 샘플(Action Units 및 감정)을 포함하는 3단 구조를 채택하며, 이 중 790개의 Event-RGB 쌍과 210개의 고정밀 정렬 쌍을 포함한다. 또한 재현 가능한 다중모달 기준선을 보고하며, 여기서 교차 모달 융합이 단일 모달리티 대비 성능을 향상시켜 이벤트 역학과 RGB 단서의 상호 보완성을 뒷받침한다. 동축 보정의 필요성을 제거함으로써, CDER-SME는 기기 간 정렬 및 실세계 정서 지능에서 배포 가능한 Event-RGB MER를 위한 실용적인 벤치마크를 제공한다.