Pose Estimation

The COCO (Common Objects in Context) dataset is a large-scale object detection, segmentation, and captioning dataset. It is designed to encourage research on a wide variety of object categories and is commonly used for benchmarking computer vision models. It is an essential dataset for researchers and developers working on object detection, segmentation, and pose estimation tasks.

KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) is one of the most popular datasets for use in mobile robotics and autonomous driving. It consists of hours of traffic scenarios recorded with a variety of sensor modalities, including high-resolution RGB, grayscale stereo cameras, and a 3D laser scanner. Despite its popularity, the dataset itself does not contain ground truth for semantic segmentation. However, various researchers have manually annotated parts of the dataset to fit their necessities. Álvarez et al. generated ground truth for 323 images from the road detection challenge with three classes: road, vertical, and sky. Zhang et al. annotated 252 (140 for training and 112 for testing) acquisitions – RGB and Velodyne scans – from the tracking challenge for ten object categories: building, sky, road, vegetation, sidewalk, car, pedestrian, cyclist, sign/pole, and fence. Ros et al. labeled 170 training images and 46 testing images (from the visual odometry challenge) with 11 classes: building, tree, sky, car, sign, road, pedestrian, fence, pole, sidewalk, and bicyclist. Source: A Review on Deep Learning Techniques Applied to Semantic Segmentation Image Source: http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d

The Human3.6M dataset is one of the largest motion capture datasets, which consists of 3.6 million human poses and corresponding images captured by a high-speed motion capture system. There are 4 high-resolution progressive scan cameras to acquire video data at 50 Hz. The dataset contains activities by 11 professional actors in 17 scenarios: discussion, smoking, taking photo, talking on the phone, etc., as well as provides accurate 3D joint positions and high-resolution videos. Source: Space-Time Representation of People Based on 3D Skeletal Data: A Review Image Source: Yu et al

The MPII Human Pose Dataset for single person pose estimation is composed of about 25K images of which 15K are training samples, 3K are validation samples and 7K are testing samples (which labels are withheld by the authors). The images are taken from YouTube videos covering 410 different human activities and the poses are manually annotated with up to 16 body joints. Source: 2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning Image Source: http://human-pose.mpi-inf.mpg.de/

The Waymo Open Dataset is comprised of high resolution sensor data collected by autonomous vehicles operated by the Waymo Driver in a wide variety of conditions. The Waymo Open Dataset currently contains 1,950 segments. The authors plan to grow this dataset in the future. Currently the datasets includes: * 1,950 segments of 20s each, collected at 10Hz (390,000 frames) in diverse geographies and conditions * Sensor data * 1 mid-range lidar * 4 short-range lidars * 5 cameras (front and sides) * Synchronized lidar and camera data * Lidar to camera projections * Sensor calibrations and vehicle poses * Labeled data * Labels for 4 object classes - Vehicles, Pedestrians, Cyclists, Signs * High-quality labels for lidar data in 1,200 segments * 12.6M 3D bounding box labels with tracking IDs on lidar data * High-quality labels for camera data in 1,000 segments * 11.8M 2D bounding box labels with tracking IDs on camera data

NTU RGB+D is a large-scale dataset for RGB-D human action recognition. It involves 56,880 samples of 60 action classes collected from 40 subjects. The actions can be generally divided into three categories: 40 daily actions (e.g., drinking, eating, reading), nine health-related actions (e.g., sneezing, staggering, falling down), and 11 mutual actions (e.g., punching, kicking, hugging). These actions take place under 17 different scene conditions corresponding to 17 video sequences (i.e., S001–S017). The actions were captured using three cameras with different horizontal imaging viewpoints, namely, −45∘,0∘, and +45∘. Multi-modality information is provided for action characterization, including depth maps, 3D skeleton joint position, RGB frames, and infrared sequences. The performance evaluation is performed by a cross-subject test that split the 40 subjects into training and test groups, and by a cross-view test that employed one camera (+45∘) for testing, and the other two cameras for training. Source: Action Recognition for Depth Video using Multi-view Dynamic Images

The 3D Poses in the Wild dataset is the first dataset in the wild with accurate 3D poses for evaluation. While other datasets outdoors exist, they are all restricted to a small recording volume. 3DPW is the first one that includes video footage taken from a moving phone camera. The dataset includes: * 60 video sequences. * 2D pose annotations. * 3D poses obtained with the method introduced in the paper. * Camera poses for every frame in the sequences. * 3D body scans and 3D people models (re-poseable and re-shapeable). Each sequence contains its corresponding models. * 18 3D models in different clothing variations. Source: https://virtualhumans.mpi Image Source: https://virtualhumans.mpi

AMASS is a large database of human motion unifying different optical marker-based motion capture datasets by representing them within a common framework and parameterization. AMASS is readily useful for animation, visualization, and generating training data for deep learning.

MPI-INF-3DHP is a 3D human body pose estimation dataset consisting of both constrained indoor and complex outdoor scenes. It records 8 actors performing 8 activities from 14 camera views. It consists on >1.3M frames captured from the 14 cameras. Source: Anatomy-aware 3D Human Pose Estimation in Videos Image Source: https://arxiv.org/abs/1611.09813

Description: 10,000 People - Human Pose Recognition Data. This dataset includes indoor and outdoor scenes.This dataset covers males and females. Age distribution ranges from teenager to the elderly, the middle-aged and young people are the majorities. The data diversity includes different shooting heights, different ages, different light conditions, different collecting environment, clothes in different seasons, multiple human poses. For each subject, the labels of gender, race, age, collecting environment and clothes were annotated. The data can be used for human pose recognition and other tasks. Data size: 10,000 people Race distribution: Asian (Chinese)

DensePose-COCO is a large-scale ground-truth dataset with image-to-surface correspondences manually annotated on 50K COCO images and train DensePose-RCNN, to densely regress part-specific UV coordinates within every human region at multiple frames per second. Source: DensePose: Dense Human Pose Estimation In The Wildc Image Source: Guler et al

JHMDB is an action recognition dataset that consists of 960 video sequences belonging to 21 actions. It is a subset of the larger HMDB51 dataset collected from digitized movies and YouTube videos. The dataset contains video and annotation for puppet flow per frame (approximated optimal flow on the person), puppet mask per frame, joint positions per frame, action label per clip and meta label per clip (camera motion, visible body parts, camera viewpoint, number of people, video quality). Source: Unsupervised Deep Metric Learning via Orthogonality based Probabilistic Loss Image Source: https://arxiv.org/pdf/1712.06316.pdf

The Pascal3D+ multi-view dataset consists of images in the wild, i.e., images of object categories exhibiting high variability, captured under uncontrolled settings, in cluttered scenes and under many different poses. Pascal3D+ contains 12 categories of rigid objects selected from the PASCAL VOC 2012 dataset. These objects are annotated with pose information (azimuth, elevation and distance to camera). Pascal3D+ also adds pose annotated images of these 12 categories from the ImageNet dataset. Source: Convolutional Models for Joint Object Categorization and Pose Estimation Image Source: Beyond PASCAL: A benchmark for 3D object detection in the wild

The Vimeo-90K is a large-scale high-quality video dataset for lower-level video processing. It proposes three different video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution. Source: https://arxiv.org/pdf/1711.09078.pdf Image Source: http://toflow.csail.mit.edu/

The 300-W is a face dataset that consists of 300 Indoor and 300 Outdoor in-the-wild images. It covers a large variation of identity, expression, illumination conditions, pose, occlusion and face size. The images were downloaded from google.com by making queries such as “party”, “conference”, “protests”, “football” and “celebrities”. Compared to the rest of in-the-wild datasets, the 300-W database contains a larger percentage of partially-occluded images and covers more expressions than the common “neutral” or “smile”, such as “surprise” or “scream”. Images were annotated with the 68-point mark-up using a semi-automatic methodology. The images of the database were carefully selected so that they represent a characteristic sample of challenging but natural face instances under totally unconstrained conditions. Thus, methods that achieve accurate performance on the 300-W database can demonstrate the same accuracy in most realistic cases. Many images of the database contain more than one annotated faces (293 images with 1 face, 53 images with 2 faces and 53 images with [3, 7] faces). Consequently, the database consists of 600 annotated face instances, but 399 unique images. Finally, there is a large variety of face sizes. Specifically, 49.3% of the faces have size in the range [48.6k, 2.0M] and the overall mean size is 85k (about 292 × 292) pixels. Source: https://ibug.doc.ic.ac.uk/media/uploads/documents/sagonas_2016_imavis.pdf Image Source: https://www.researchgate.net/profile/Xuanyi_Dong/publication/323722412/figure/fig1/AS:679426136227845@1538999222829/Face-samples-from-300-W-dataset-Different-faces-have-different-styles-whereas-the-style_Q640.jpg

The Leeds Sports Pose (LSP) dataset is widely used as the benchmark for human pose estimation. The original LSP dataset contains 2,000 images of sportspersons gathered from Flickr, 1000 for training and 1000 for testing. Each image is annotated with 14 joint locations, where left and right joints are consistently labelled from a person-centric viewpoint. The extended LSP dataset contains additional 10,000 images labeled for training. Source: Deep Deformation Network for Object Landmark Localization Image: Sumer et al

The PASCAL Visual Object Classes (VOC) 2012 dataset contains 20 object categories including vehicles, household, animals, and other: aeroplane, bicycle, boat, bus, car, motorbike, train, bottle, chair, dining table, potted plant, sofa, TV/monitor, bird, cat, cow, dog, horse, sheep, and person. Each image in this dataset has pixel-level segmentation annotations, bounding box annotations, and object class annotations. This dataset has been widely used as a benchmark for object detection, semantic segmentation, and classification tasks. The PASCAL VOC dataset is split into three subsets: 1,464 images for training, 1,449 images for validation and a private testing set. Source: Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey Image Source: http://host.robots.ox.ac.uk/pascal/VOC/voc2012/examples/images/sheep_06.jpg

The YCB-Video dataset is a large-scale video dataset for 6D object pose estimation. provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames. Source: https://rse-lab.cs.washington.edu/projects/posecnn/ Image Source: https://www.researchgate.net/figure/Examples-of-refined-poses-on-the-YCB-Video-dataset-which-use-results-from-PoseCNN-Xiang_fig6_339663565

The Annotated Facial Landmarks in the Wild (AFLW) is a large-scale collection of annotated face images gathered from Flickr, exhibiting a large variety in appearance (e.g., pose, expression, ethnicity, age, gender) as well as general imaging and environmental conditions. In total about 25K faces are annotated with up to 21 landmarks per image. Source: Nose, Eyes and Ears: Head Pose Estimation by Locating Facial Keypoints

MPII Human Pose Dataset is a dataset for human pose estimation. It consists of around 25k images extracted from online videos. Each image contains one or more people, with over 40k people annotated in total. Among the 40k samples, ∼28k samples are for training and the remainder are for testing. Overall the dataset covers 410 human activities and each image is provided with an activity label. Images were extracted from a YouTube video and provided with preceding and following un-annotated frames. Source: Accelerating Large-Kernel Convolution Using Summed-Area Tables Image Source: http://human-pose.mpi-inf.mpg.de/

The Pix3D dataset is a large-scale benchmark of diverse image-shape pairs with pixel-level 2D-3D alignment. Pix3D has wide applications in shape-related tasks including reconstruction, retrieval, viewpoint estimation, etc. Source: http://pix3d.csail.mit.edu/ Image Source: http://pix3d.csail.mit.edu/

CMU Panoptic is a large scale dataset providing 3D pose annotations (1.5 millions) for multiple people engaging social activities. It contains 65 videos (5.5 hours) with multi-view annotations, but only 17 of them are in multi-person scenario and have the camera parameters. Massively Multiview System * 480 VGA camera views * 30+ HD views * 10 RGB-D sensors * Hardware-based sync * Calibration * Interesting Scenes with Labels Multiple people * Socially interacting groups * 3D body pose * 3D facial landmarks * Transcripts + speaker ID Hardware setup * 480 VGA cameras, 640 x 480 resolution, 25 fps, synchronized among themselves using a hardware clock * 31 HD cameras, 1920 x 1080 resolution, 30 fps, synchronized among themselves using a hardware clock, timing aligned with VGA cameras * 10 Kinect Ⅱ Sensors. 1920 x 1080 (RGB), 512 x 424 (depth), 30 fps, timing aligned among themselves and other sensors 5 DLP Projectors. synchronized with HD cameras Source: Single-Stage Multi-Person Pose Machines Image Source: http://domedb.perception.cs.cmu.edu/

The Labeled Face Parts in-the-Wild (LFPW) consists of 1,432 faces from images downloaded from the web using simple text queries on sites such as google.com, flickr.com, and yahoo.com. Each image was labeled by three MTurk workers, and 29 fiducial points, shown below, are included in dataset. Source: https://neerajkumar.org/databases/lfpw/ Image Source: https://neerajkumar.org/databases/lfpw/

SUN3D contains a large-scale RGB-D video database, with 8 annotated sequences. Each frame has a semantic segmentation of the objects in the scene and information about the camera pose. It is composed by 415 sequences captured in 254 different spaces, in 41 different buildings. Moreover, some places have been captured multiple times at different moments of the day. Source: A Review on Deep Learning TechniquesApplied to Semantic Segmentation Image Source: http://sun3d.cs.princeton.edu/

FreiHAND is a 3D hand pose dataset which records different hand actions performed by 32 people. For each hand image, MANO-based 3D hand pose annotations are provided. It currently contains 32,560 unique training samples and 3960 unique samples for evaluation. The training samples are recorded with a green screen background allowing for background removal. In addition, it applies three different post processing strategies to training samples for data augmentation. However, these post processing strategies are not applied to evaluation samples. Source: Knowledge as Priors: Cross-Modal Knowledge Generalizationfor Datasets without Superior Knowledge Image Source: https://lmb.informatik.uni-freiburg.de/resources/datasets/FreihandDataset.en.html

The NCLT dataset is a large scale, long-term autonomy dataset for robotics research collected on the University of Michigan’s North Campus. The dataset consists of omnidirectional imagery, 3D lidar, planar lidar, GPS, and proprioceptive sensors for odometry collected using a Segway robot. The dataset was collected to facilitate research focusing on long-term autonomous operation in changing environments. The dataset is comprised of 27 sessions spaced approximately biweekly over the course of 15 months. The sessions repeatedly explore the campus, both indoors and outdoors, on varying trajectories, and at different times of the day across all four seasons. This allows the dataset to capture many challenging elements including: moving obstacles (e.g., pedestrians, bicyclists, and cars), changing lighting, varying viewpoint, seasonal and weather changes (e.g., falling leaves and snow), and long-term structural changes caused by construction projects. Source: http://robots.engin.umich.edu/nclt/nclt.pdf Image Source: http://robots.engin.umich.edu/nclt/

AFLW2000-3D is a dataset of 2000 images that have been annotated with image-level 68-point 3D facial landmarks. This dataset is used for evaluation of 3D facial landmark detection models. The head poses are very diverse and often hard to be detected by a CNN-based face detector. Source: https://www.tensorflow.org/datasets/catalog/aflw2k3d Image Source: http://www.cbsr.ia.ac.cn/users/xiangyuzhu/projects/3DDFA/main.htm

The Caltech Occluded Faces in the Wild (COFW) dataset is designed to present faces in real-world conditions. Faces show large variations in shape and occlusions due to differences in pose, expression, use of accessories such as sunglasses and hats and interactions with objects (e.g. food, hands, microphones, etc.). All images were hand annotated using the same 29 landmarks as in LFPW. Both the landmark positions as well as their occluded/unoccluded state were annotated. The faces are occluded to different degrees, with large variations in the type of occlusions encountered. COFW has an average occlusion of over 23. Source: http://www.vision.caltech.edu/xpburgos/ICCV13/#dataset Image Source: http://www.vision.caltech.edu/xpburgos/ICCV13/#dataset

The Penn Action Dataset contains 2326 video sequences of 15 different actions and human joint annotations for each sequence. Source: http://dreamdragon.github.io/PennAction/ Image Source: http://dreamdragon.github.io/PennAction/

For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. Hypersim is a photorealistic synthetic dataset for holistic indoor scene understanding. It contains 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry. Source: https://github.com/apple/ml-hypersim Image Source: https://github.com/apple/ml-hypersim

The Wider Facial Landmarks in the Wild or WFLW database contains 10000 faces (7500 for training and 2500 for testing) with 98 annotated landmarks. This database also features rich attribute annotations in terms of occlusion, head pose, make-up, illumination, blur and expressions. Source: Deep Entwined Learning Head Pose and Face Alignment Inside an Attentional Cascade with Doubly-Conditional fusion Image Source: https://wywu.github.io/projects/LAB/WFLW.html

The PoseTrack dataset is a large-scale benchmark for multi-person pose estimation and tracking in videos. It requires not only pose estimation in single frames, but also temporal tracking across frames. It contains 514 videos including 66,374 frames in total, split into 300, 50 and 208 videos for training, validation and test set respectively. For training videos, 30 frames from the center are annotated. For validation and test videos, besides 30 frames from the center, every fourth frame is also annotated for evaluating long range articulated tracking. The annotations include 15 body keypoints location, a unique person id and a head bounding box for each person instance. Source: Simple Baselines for Human Pose Estimation and Tracking Image Source: https://posetrack.net/

The CrowdPose dataset contains about 20,000 images and a total of 80,000 human poses with 14 labeled keypoints. The test set includes 8,000 images. The crowded images containing homes are extracted from MSCOCO, MPII and AI Challenger. Source: Human Pose Estimation for Real-World Crowded Scenarios Image Source: https://github.com/Jeff-sjtu/CrowdPose

DexYCB is a dataset for capturing hand grasping of objects. It can be used three relevant tasks: 2D object and keypoint detection, 6D object pose estimation, and 3D hand pose estimation. The dataset was built using 20 objects from the YCB-Video dataset, and consists of multiple trials from 10 subjects. For each trial, there is a target object with 2 to 4 other objects placed on a table. The subject is asked to start from a relaxed pose, pick up the target object, and hold it in the air. Some subjects were asked to pretend to hand over the object to someone across from them. Each action is recorded for 3 seconds, repeating the trial 5 times for each target object, each time with a random set of accompanied objects and placement. In total there are 100 trials per subject, and 1,000 trials in total for all subjects. Source: DexYCB: A Benchmark for Capturing Hand Grasping of Objects

T-LESS is a dataset for estimating the 6D pose, i.e. translation and rotation, of texture-less rigid objects. The dataset features thirty industry-relevant objects with no significant texture and no discriminative color or reflectance properties. The objects exhibit symmetries and mutual similarities in shape and/or size. Compared to other datasets, a unique property is that some of the objects are parts of others. The dataset includes training and test images that were captured with three synchronized sensors, specifically a structured-light and a time-of-flight RGB-D sensor and a high-resolution RGB camera. There are approximately 39K training and 10K test images from each sensor. Additionally, two types of 3D models are provided for each object, i.e. a manually created CAD model and a semi-automatically reconstructed one. Training images depict individual objects against a black background. Test images originate from twenty test scenes having varying complexity, which increases from simple scenes with several isolated objects to very challenging ones with multiple instances of several objects and with a high amount of clutter and occlusion. The images were captured from a systematically sampled view sphere around the object/scene, and are annotated with accurate ground truth 6D poses of all modeled objects. Source: http://cmp.felk.cvut.cz/t-less/ Image Source: http://cmp.felk.cvut.cz/t-less/

Aachen Day-Night is a dataset designed for benchmarking 6DOF outdoor visual localization in changing conditions. It focuses on localizing high-quality night-time images against a day-time 3D model. There are 14,607 images with changing conditions of weather, season and day-night cycles. Source: Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions