Human Interaction Recognition

NTU RGB+D is a large-scale dataset for RGB-D human action recognition. It involves 56,880 samples of 60 action classes collected from 40 subjects. The actions can be generally divided into three categories: 40 daily actions (e.g., drinking, eating, reading), nine health-related actions (e.g., sneezing, staggering, falling down), and 11 mutual actions (e.g., punching, kicking, hugging). These actions take place under 17 different scene conditions corresponding to 17 video sequences (i.e., S001–S017). The actions were captured using three cameras with different horizontal imaging viewpoints, namely, −45∘,0∘, and +45∘. Multi-modality information is provided for action characterization, including depth maps, 3D skeleton joint position, RGB frames, and infrared sequences. The performance evaluation is performed by a cross-subject test that split the 40 subjects into training and test groups, and by a cross-view test that employed one camera (+45∘) for testing, and the other two cameras for training. Source: Action Recognition for Depth Video using Multi-view Dynamic Images

NTU RGB+D 120 is a large-scale dataset for RGB+D human action recognition, which is collected from 106 distinct subjects and contains more than 114 thousand video samples and 8 million frames. This dataset contains 120 different action classes including daily, mutual, and health-related activities. Source: NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding

SBU-Kinect-Interaction dataset version 2.0 comprises of RGB-D video sequences of humans performing interaction activities that are recording using the Microsoft Kinect sensor. This dataset was originally recorded for a class project, and it must be used only for the purposes of research. If you use this dataset in your work, please cite the following paper. Kiwon Yun, Jean Honorio, Debaleena Chattopadhyay, Tamara L. Berg, and Dimitris Samaras, The 2nd International Workshop on Human Activity Understanding from 3D Data at Conference on Computer Vision and Pattern Recognition (HAU3D-CVPRW), CVPR 2012 SBU-Refine: SBU-Refine relabels the test set manually and refines the noise labels in training set by algorithm. H. Yang, T. Wang, X. Hu, and C.-W. Fu, “SILT: Shadow-aware iterative label tuning for learning to detect shadows from noisy labels,” in ICCV, 2023, pp. 12 687–12 698.

The UT-Interaction dataset contains videos of continuous executions of 6 classes of human-human interactions: shake-hands, point, hug, push, kick and punch. Ground truth labels for these interactions are provided, including time intervals and bounding boxes. There is a total of 20 video sequences whose lengths are around 1 minute. Each video contains at least one execution per interaction, resulting in 8 executions of human activities per video on average. Several participants with more than 15 different clothing conditions appear in the videos. The videos are taken with the resolution of 720*480, 30fps, and the height of a person in the video is about 200 pixels. Source: https://cvrc.ece.utexas.edu/SDHA2010/Human_Interaction.html Image Source: https://cvrc.ece.utexas.edu/SDHA2010/Human_Interaction.html

EPIC-SOUNDS is a large scale dataset of audio annotations capturing temporal extents and class labels within the audio stream of the egocentric videos from EPIC-KITCHENS-100. EPIC-SOUNDS includes 78.4k categorised and 39.2k non-categorised segments of audible events and actions, distributed across 44 classes. Source: Epic-Sounds: A Large-scale Dataset of Actions That Sound

3D dataset of humans Manipulating Objects in-the-Wild (MOW). The dataset contains 512 images in the wild, spanning 121 object categories with annotation of instance category, 3D object models, 3D hand pose, and object pose annotation.

The Human-to-Human-or-Object Interaction Dataset (H2O) dataset is a dataset for Human-Object Interaction (HOI) detection. It consists in determining and locating the list of triplets <subject,verb,target> which describe all the simultaneous interactions in an image. H²O is composed of the 10 301 images from V-COCO dataset to which are added 3 635 images which mostly contain interactions between people.

The NuiSI dataset contains skeleton tracking trajectories of Human Interaction Partners performing a variety of physically interactive behaviors (waving, handshaking, rocket fistbump, parachute fistbump) with each other. This is inspired by the dataset in Bütepage et al. "Imitating by generating: Deep generative models for imitation of interactive tasks." Frontiers in Robotics and AI (2020) wherein they capture a dataset with rokoko motion capture suits. Instead we track the skeletons of the interaction partner with Intel Realsense cameras using Nuitrack, for a more realistic scenario, with noise coming from the depth sensor, the skeleton tracking and some partial occlusions. This makes it more representative of real world interactions with a Robot equipped with an RGBD camera. T This dataset is used in our papers for training Interaction models for Human-Robot Interaction with a humanoid social robot. If you find the dataset useful in your work, please cite our paper: Prasad, V., Heitlinger, L., Koert, D., Stock-Homburg, R., Peters, J., & Chalvatzaki, G. (2023). Learning multimodal latent dynamics for human-robot interaction. arXiv preprint arXiv:2311.16380. @article{prasad2023learning, title={Learning multimodal latent dynamics for human-robot interaction}, author={Prasad, Vignesh and Heitlinger, Lea and Koert, Dorothea and Stock-Homburg, Ruth and Peters, Jan and Chalvatzaki, Georgia}, journal={arXiv preprint arXiv:2311.16380}, year={2023} }

H²O is an image dataset annotated for Human-to-human-or-object interaction detection. H²O is composed of the images from V-COCO dataset to which are added images which mostly contain interactions between people. The dataset has been introduced in this paper: Orcesi, A., Audigier, R., Toukam, F. P., & Luvison, B. (2021, December). Detecting Human-to-Human-or-Object (H 2 O) Interactions with DIABOLO. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (pp. 1-8). IEEE. The annotations were made with Pixano, an opensource, smart annotation tool for computer vision applications: https://pixano.cea.fr/