Multimodal Emotion Recognition

Multimodal Emotion Recognition IEMOCAP The IEMOCAP dataset consists of 151 videos of recorded dialogues, with 2 speakers per session for a total of 302 videos across the dataset. Each segment is annotated for the presence of 9 emotions (angry, excited, fear, sad, surprised, frustrated, happy, disappointed and neutral) as well as valence, arousal and dominance. The dataset is recorded across 5 sessions with 5 pairs of speakers. Source: Multi-attention Recurrent Network for Human Communication Comprehension Image Source: https://sail.usc.edu/iemocap/Busso_2008_iemocap.pdf

Multimodal EmotionLines Dataset (MELD) has been created by enhancing and extending EmotionLines dataset. MELD contains the same dialogue instances available in EmotionLines, but it also encompasses audio and visual modality along with text. MELD has more than 1400 dialogues and 13000 utterances from Friends TV series. Multiple speakers participated in the dialogues. Each utterance in a dialogue has been labeled by any of these seven emotions -- Anger, Disgust, Sadness, Joy, Neutral, Surprise and Fear. MELD also has sentiment (positive, negative and neutral) annotation for each utterance. Source: https://affective-meld.github.io/ Image Source: https://affective-meld.github.io/

CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) is the largest dataset of sentence-level sentiment analysis and emotion recognition in online videos. CMU-MOSEI contains over 12 hours of annotated video from over 1000 speakers and 250 topics.

The EMOTIC dataset, named after EMOTions In Context, is a database of images with people in real environments, annotated with their apparent emotions. The images are annotated with an extended list of 26 emotion categories combined with the three common continuous dimensions Valence, Arousal and Dominance. Source: Context Based Emotion Recognition using EMOTIC Dataset

We construct a dataset named CPED from 40 Chinese TV shows. CPED consists of multisource knowledge related to empathy and personal characteristic. This knowledge covers 13 emotions, gender, Big Five personality traits, 19 dialogue acts and other knowledge. * We build a multiturn Chinese Personalized and Emotional Dialogue dataset called CPED. To the best of our knowledge, CPED is the first Chinese personalized and emotional dialogue dataset. CPED contains 12K dialogues and 133K utterances with multi-modal context. Therefore, it can be used in both complicated dialogue understanding and human-like conversation generation. * CPED has been annotated with 3 character attributes (name, gender age), Big Five personality traits, 2 types of dynamic emotional information (sentiment and emotion) and DAs. The personality traits and emotions can be used as prior external knowledge for open-domain conversation generation, making the conversation system have a good command of personification capabilities. * We propose three tasks for CPED: personality recognition in conversations (PRC), emotion recognition in conversations (ERC), and personalized and emotional conversation (PEC). A set of experiments verify the importance of using personalities and emotions as prior external knowledge for conversation generation.

The DEAP dataset consists of two parts: - The ratings from an online self-assessment where 120 one-minute extracts of music videos were each rated by 14-16 volunteers based on arousal, valence and dominance. - The participant ratings, physiological recordings and face video of an experiment where 32 volunteers watched a subset of 40 of the above music videos. EEG and physiological signals were recorded and each participant also rated the videos as above. For 22 participants frontal face video was also recorded.

the YF-E6 emotion dataset using the 6 basic emotion type as keywords on social video-sharing websites including YouTube and Flickr, leading to a total of 3000 videos. The dataset is labeled through crowdsourcing by 10 different annotators (5 males and 5 females), whose age ranged from 22 to 45. Annotators were given detailed definition for each emotion before performing the task. Every video is manually labeled by all the annotators. A video is excluded from the final dataset when over half of annotations are inconsistent with the initial search keyword.

Russian dataset of emotional speech dialogues. This dataset was assembled from ~3.5 hours of live speech by actors who voiced pre-distributed emotions in the dialogue for ~3 minutes each. <br> Each sample of dataset contains name of part from the original dataset studio source, speech file (16000 or 44100Hz) of human voice, 1 of 7 labeled emotions and the speech-to-texted part of voice speech. <br> Emotions are represented in 7 states: anger, disgust, fear, enthusiasm, happiness, neutral and sadness. This dataset was created by Artem Amentes, Nikita Davidchuk and Ilya Lubenets @misc{Aniemore, author = {Артем Аментес, Илья Лубенец, Никита Давидчук}, title = {Открытая библиотека искусственного интеллекта для анализа и выявления эмоциональных оттенков речи человека}, year = {2022}, publisher = {Hugging Face}, journal = {Hugging Face Hub}, howpublished = {\url{https://huggingface.com/aniemore/Aniemore}}, email = {hello@socialcode.ru} }

The development of ecologically valid procedures for collecting reliable and unbiased emotional data towards computer interfaces with social and affective intelligence targeting patients with mental disorders. Following its development, presented with, the Athens Emotional States Inventory (AESI) proposes the design, recording and validation of an audiovisual database for five emotional states: anger, fear, joy, sadness and neutral. The items of the AESI consist of sentences each having content indicative of the corresponding emotion. Emotional content was assessed through a survey of 40 young participants with a questionnaire following the Latin square design. The emotional sentences that were correctly identified by 85% of the participants were recorded in a soundproof room with microphones and cameras. A preliminary validation of AESI is performed through automatic emotion recognition experiments from speech. The resulting database contains 696 recorded utterances in Greek language by 20 native speakers and has a total duration of approximately 28 min. Speech classification results yield accuracy up to 75.15% for automatically recognizing the emotions in AESI. These results indicate the usefulness of our approach for collecting emotional data with reliable content, balanced across classes and with reduced environmental variability.

Currently, an essential point in speech synthesis is the addressing of the variability of human speech. One of the main sources of this diversity is the emotional state of the speaker. Most of the recent work in this area has been focused on the prosodic aspects of speech and on rule-based formant synthesis experiments. Even when adopting an improved voice source, we cannot achieve a smiling happy voice or the menacing quality of cold anger. For this reason, we have performed two experiments aimed at developing a concatenative emotional synthesiser, a synthesiser that can copy the quality of an emotional voice without an explicit mathematical model.

Existing databases usually record posed or induced human behavior in individual or dyadic settings with biased annotations in which a basic emotional class label or a Valence–Arousal pair value represents the emotional states. To address this need,We created the “Werewolf-XL” database, which contains a total of 987 h of spontaneous audio-visual recordings of 129 subjects in a group interaction of nine individuals playing a conversational role-playing game called Werewolf. We provide 133,461 individual utterance-level video clips with internal (18 non-prototypical emotional categories) and external (pleasure, arousal, and dominance) emotion annotations including 14,635 samples from speaking players. Besides, the results of annotation agreement analysis show fair reliability and validity. Furthermore, we provided extensive benchmarks of unimodal and multimodal emotional recognition results. The database (the video clips, annotations, features and protocols) is made publicly available, which provides great opportunities for researchers in affective computing and beyond.

Welcome to L-SVD L-SVD is an extensive and rigorously curated video dataset aimed at transforming the field of emotion recognition. This dataset features more than 20,000 short video clips, each carefully annotated to represent a range of human emotions. L-SVD stands at the intersection of Cognitive Science, Psychology, Computer Science, and Medical Science, providing a unique tool for both research and application in these fields. ### Why L-SVD? Inspired by the revolutionary ImageNet, L-SVD aspires to be a benchmark in the emotional AI space, offering meticulously labeled and consistently processed video data to the global research community. - Rich Emotional Annotations: Our dataset encompasses a spectrum of eight emotions—Anger, Contempt, Disgust, Enjoyment, Fear, Sadness, Surprise, and Neutral—each annotated with precision. - Uniform Video Quality: We ensure uniform hue, contrast, and brightness across all videos to provide a standard quality baseline for algorithm development. - Exponential Growth and Community Driven: L-SVD is rapidly expanding, with new contributions regularly enhancing the depth and breadth of the dataset. ### Dataset Features - Detailed Emotional Range: L-SVD covers a broad array of emotional states, meticulously labeled for accurate recognition. - Pre-Processed for Excellence: All clips are pre-processed to ensure high fidelity and uniformity, setting a high standard for training data quality. - Inviting Global Contributions: We encourage contributions from researchers and practitioners around the world to help grow and diversify the dataset.

The Storytelling Video Dataset is a high-quality, human-reviewed multimodal dataset featuring over 700 full-body video recordings of native Russian speakers. Each video is 10+ minutes long and includes synchronized speech, facial expressions, gestures, and emotional variation. The dataset is ideal for research and development in: 🗣️ Speech-to-text & voice modeling (Russian) 😃 Emotion & gesture recognition 🤖 Multimodal learning & LLM alignment 🧍 Avatar generation & digital human training Key Features: Full-body framing (waist-up or knee-up) Clear facial expressions and gestures High-quality speech audio in Russian Natural storytelling with emotional diversity Preview (10 Participants): We’ve prepared a short compilation of 10 different speakers from the dataset. Each clip is taken from a unique 10-minute unscripted video, featuring full-body framing, gestures, and emotional speech. 📺 Watch the sample: https://drive.google.com/drive/folders/1QtC2il-Qb62nZNJlOtG8WvOf-SSkSGM-?usp=drive_link – Video Preview Dataset Pages: 🐙 GitHub: github.com/MaratDV 📊 DataHub: datahub.io/@MaratDV/storytelling-video-dataset email chinzad@gmail.com or telegram @Marat_DV