Pose Estimation

BRACE is a dataset for audio-conditioned dance motion synthesis challenging common assumptions for this task: - strong music-dance correlation - controlled motion data - simple poses and movements To address these issues: - We focus on breakdancing which features acrobatic moves, tangled postures and weaker dance-music correlation. - We adopt a hybrid labelling pipeline leveraging estimation models as well as manual annotations to obtain good quality keypoint sequences at a reduced cost. - Our efforts produced the BRACE dataset, which contains over 3 hours and 30 minutes of densely annotated poses. - BRACE is also useful to fine-tune pose-estimation models thanks to its high quality keypoint annotations for complicated and uncommon poses.

The DREAM dataset is introduce by the paper "Camera-to-Robot Pose Estimation from a Single Image" (ICRA 2020). This dataset consists of synthetic images (both with and without domain randomlization) of three different robot manipulators (Franka Emika’s Panda, Kuka’s LBR iiwa 7 R800, and Rethink Robotics’ Baxter) , as well as real-world images of Franka Emika’s Panda taken from various RGBD cameras (XBox 360 Kinect (XK), RealSense (RS), and Azure Kinect (AK)). Each instance in the dataset contains an RGB image, keypoint 3D/2D coordinates , global camera-to-robot transformation and joint state configurations (from both revolute and prismatic joint) of the robot. Tasks like estimating robot pose (camera pose) from a single RGB image, camera-to-robot calibration can be conducted and evaluated in this dataset.

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

We introduce HARPER, a novel dataset for 3D body pose estimation and forecast in dyadic interactions between users and \spot, the quadruped robot manufactured by Boston Dynamics. The key-novelty is the focus on the robot's perspective, i.e., on the data captured by the robot's sensors. These make 3D body pose analysis challenging because being close to the ground captures humans only partially. The scenario underlying HARPER includes 15 actions, of which 10 involve physical contact between the robot and users. The Corpus contains not only the recordings of the built-in stereo cameras of Spot, but also those of a 6-camera OptiTrack system (all recordings are synchronized). This leads to ground-truth skeletal representations with a precision lower than a millimeter. In addition, the Corpus includes reproducible benchmarks on 3D Human Pose Estimation, Human Pose Forecasting, and Collision Prediction, all based on publicly available baseline approaches. This enables future HARPER users to rigorously compare their results with those we provide in this work. Source: [Download dataset] (https://github.com/intelligolabs/HARPER)

We present a large-scale dataset for 3D urban scene understanding. Compared to existing datasets, our dataset consists of 75 outdoor urban scenes with diverse backgrounds, encompassing over 15,000 images. These scenes offer 360◦ hemispherical views, capturing diverse foreground objects illuminated under various lighting conditions. Additionally, our dataset encompasses scenes that are not limited to forward-driving views, addressing the limitations of previous datasets such as limited overlap and coverage between camera views. The closest pre-existing dataset for generalizable evaluation is DTU [2] (80 scenes) which comprises mostly indoor objects and does not provide multiple foreground objects or background scenes. We use the Parallel Domain synthetic data generation to render high-fidelity 360◦ scenes. We select 3 different maps i.e. SF 6thAndMission, SF GrantAndCalifornia and SF VanNessAveAndTurkSt and sample 75 different scenes across all 3 maps as our backgrounds (All 75 scenes across 3 maps are significantly different road scenes from each other, captured at different viewpoints in the city). We select 20 different cars in 50 different textures for training and randomly sample from 1 to 4 cars to render in a scene. We refer to this dataset as NeRDS 360: NeRF for Reconstruction, Decomposition and Scene Synthesis of 360◦ outdoor scenes. In total, we generate 15k renderings by sampling 200 cameras in a hemispherical dome at a fixed distance from the center of cars. We held out 5 scenes with 4 different cars and different backgrounds for testing, comprising 100 cameras distributed uniformly sampled in the upper hemisphere, different from the camera distributions used for training. We use the diverse validation camera distribution to test our approach’s ability to generalize to unseen viewpoints as well as unseen scenes during training. Our dataset and the corresponding task is extremely challenging due to occlusions, diversity of backgrounds, and rendered objects with various lightning and shadows. Our task entails reconstructing 360◦ hemispherical views of complete scenes using a handful of observations i.e. 1 to 5 as shown by red cameras whereas evaluating using all 100 hemispherical views, hence our task requires strong priors for novel view synthesis of outdoor scenes. Tasks our dataset support: - Generaliazable Novel view synthesis (Few shot evaluation) - Novel view synthesis (Overfitting evaluation) - 6D pose estimation - Object editing - Depth estimation - Semantic Segmentation - Instance Segmentation

PedX is a large-scale multi-modal collection of pedestrians at complex urban intersections. The dataset provides high-resolution stereo images and LiDAR data with manual 2D and automatic 3D annotations. The data was captured using two pairs of stereo cameras and four Velodyne LiDAR sensors. Source: pedx.io

AwA Pose is a large scale animal keypoint dataset with ground truth annotations for keypoint detection of quadruped animals from images.

The Few-Shot Object Learning (FewSOL) dataset can be used for object recognition with a few images per object. It contains 336 real-world objects with 9 RGB-D images per object from different views. Object segmentation masks, object poses and object attributes are provided. In addition, synthetic images generated using 330 3D object models are used to augment the dataset. FewSOL dataset can be used to study a set of few-shot object recognition problems such as classification, detection and segmentation, shape reconstruction, pose estimation, keypoint correspondences and attribute recognition. Motivation: If robots can recognize objects from a few exemplar images, it is possible to scale up the number of objects a robot can recognize because collecting a few images per object is a much easier process compared to building a 3D model of an object. In addition, models trained in the meta-learning setting can generalize to new objects without re-training.

The Fraunhofer IPA Bin-Picking dataset is a large-scale dataset comprising both simulated and real-world scenes for various objects (potentially having symmetries) and is fully annotated with 6D poses. A pyhsics simulation is used to create scenes of many parts in bulk by dropping objects in a random position and orientation above a bin. Additionally, this dataset extends the Siléane dataset by providing more samples. This allows to e.g. train deep neural networks and benchmark the performance on the public Siléane dataset Source: https://www.bin-picking.ai/en/dataset.html Image Source: https://arxiv.org/abs/1912.12125

These images were generated using Blender and IEE-Simulator with different head-poses, where the images are labelled according to nine classes (straight, turned bottom-left, turned left, turned top-left, turned bottom-right, turned right, turned top-right, reclined, looking up). The dataset contains 16,013 training images and 2,825 testing images, in addition to 4,700 images for improvements.

HuPR is a human pose estimation benchmark is created using cross-calibrated mmWave radar sensors and a monocular RGB camera for cross-modality training of radar-based human pose estimation. This dataset contains 235 sequences of data in an indoor environment, with each sequence being one-minute long and totalling about 4 hour-long video data. Source: HuPR: A Benchmark for Human Pose Estimation Using Millimeter Wave Radar Image Source: https://arxiv.org/pdf/2210.12564v1.pdf

The MERL-RAV (MERL Reannotation of AFLW with Visibility) Dataset contains over 19,000 face images in a full range of head poses. Each face is manually labeled with the ground-truth locations of 68 landmarks, with the additional information of whether each landmark is unoccluded, self-occluded (due to extreme head poses), or externally occluded. The images were annotated by professional labelers, supervised by researchers at Mitsubishi Electric Research Laboratories (MERL).

Our RealMCVSR dataset provides real-world HD video triplets concurrently recorded by Apple iPhone 12 Pro Max equipped with triple cameras having fixed focal lengths: ultra-wide (30mm), wide-angle (59mm), and telephoto (147mm). To concurrently record video triplets, we built an iOS app that provides full control over exposure parameters (i.e., shutter speed and ISO) of the cameras. For recording each scene, we set the cameras in the auto-exposure mode, where the shutter speeds of the three cameras are synced to avoid varying motion blur across a video triplet. ISOs are adjusted accordingly for each camera to pick up the same exposure. Each video is saved in the MOV format using HEVC/H.265 encoding with the HD resolution (1080 x 1920). The dataset contains triplets of 161 video clips with 23,107 frames in total. The video triplets are split into training, validation, and testing sets, each of which has 137, 8, and 16 triplets with 19,426, 1,141, and 2,540 frames, respectively.

Relative Human (RH) contains multi-person in-the-wild RGB images with rich human annotations, including: Depth layers: relative depth relationship/ordering between all people in the image. Age group classfication: adults, teenagers, kids, babies. Others: Genders, Bounding box, 2D pose.

Accurate 3D human pose estimation is essential for sports analytics, coaching, and injury prevention. However, existing datasets for monocular pose estimation do not adequately capture the challenging and dynamic nature of sports movements. In response, we introduce SportsPose, a large-scale 3D human pose dataset consisting of highly dynamic sports movements. With more than 176,000 3D poses from 24 different subjects performing 5 different sports activities, SportsPose provides a diverse and comprehensive set of 3D poses that reflect the complex and dynamic nature of sports movements. Contrary to other markerless datasets we have quantitatively evaluated the precision of SportsPose by comparing our poses with a commercial marker-based system and achieve a mean error of 34.5 mm across all evaluation sequences. This is comparable to the error reported on the commonly used 3DPW dataset. We further introduce a new metric, local movement, which describes the movement of the wrist and ankle joints in relation to the body. With this, we show that SportsPose contains more movement than the Human3.6M and 3DPW datasets in these extremum joints, indicating that our movements are more dynamic. The dataset with accompanying code can be downloaded from our website. We hope that SportsPose will allow researchers and practitioners to develop and evaluate more effective models for the analysis of sports performance and injury prevention. With its realistic and diverse dataset, SportsPose provides a valuable resource for advancing the state-of-the-art in pose estimation in sports

The dataset is designed specifically to solve a range of computer vision problems (2D-3D tracking, posture) faced by biologists while designing behavior studies with animals. Typically, datasets for animal-specific vision tasks are created using open-source video material. This might be effective for an initial start, but these methods are not deployment ready for the behavior community. Therefore, we designed a semi-automated method for biologists to create well-curated datasets at a large scale for the ML and Vision community. 3D-POP is the first dataset with 3D ground truth for multi-animal, multi-view tracking problems. Highlight: The dataset is captured with the intention of using it for various vision problems and with different levels of complexity (no of cameras, no of individuals) Video explanation: Link to YouTube video Video teaser: Link to YouTube video ## Dataset Features: ### Marker-based videos: - 6 hours+ of annotations of 18 individuals (groups of 1, 2, 5, 10). - Bounding box - Trajectories (2D and 3D) - Posture (2D and 3D) with 9 key points - Identities - Total of 57 sequences (4K) with 4 views. - Dataset customization* (Users can modify the dataset and add key points to the dataset) ### Markerless: - 1Hr+ videos of 18 individuals in groups of 1, 2, 5, 11. The birds have no markers. This data is provided as test cases and unsupervised approaches. ## Problems: ### 2D domain: - Position, Posture of birds (different group sizes n = 1, 2, 5, 10) with Single/Multiview. - Tracking with single - multiview ### 3D domain: - Position, Posture of birds (different group sizes n = 1, 2, 5, 10) with Single/Multiview. - Tracking with single - multiview ### Fine-grained recognition: - Identity tracking with ground truth. ### Unsupervised learning: - 2D or 3D posture problems ## Idea: The dataset is created with a motion capture system, using the 6-DOF tracking ability. Assumptions are that head and body act as rigid bodies when birds walk and forage (proved with experiment). Therefore, we get the 3D position of key points by tracking head/body orientation.

Robot grasping is often formulated as a learning problem. With the increasing speed and quality of physics simulations, generating large-scale grasping data sets that feed learning algorithms is becoming more and more popular. An often overlooked question is how to generate the grasps that make up these data sets. In this paper, we review, classify, and compare different grasp sampling strategies. Our evaluation is based on a fine-grained discretization of SE(3) and uses physics-based simulation to evaluate the quality and robustness of the corresponding parallel-jaw grasps. Specifically, we consider more than 1 billion grasps for each of the 21 objects from the YCB data set. This dense data set lets us evaluate existing sampling schemes w.r.t. their bias and efficiency. Our experiments show that some popular sampling schemes contain significant bias and do not cover all possible ways an object can be grasped.

Amateur Drawings is a dataset collected via the public demo of Animated Drawings, containing over 178,000 amateur drawings and corresponding user-accepted character bounding boxes, segmentation masks, and joint location annotations. Source: A Method for Animating Children's Drawings of the Human Figure Image Source: Animated Drawings | Github Repo

CHAIRS is a large-scale motion-captured f-AHOI dataset, consisting of 17.3 hours of versatile interactions between 46 participants and 81 articulated and rigid sittable objects. CHAIRS provides 3D meshes of both humans and articulated objects during the entire interactive process, as well as realistic and physically plausible full-body interactions. Source: CHAIRS: Towards Full-Body Articulated Human-Object Interaction Image Source: https://arxiv.org/pdf/2212.10621v1.pdf

The Composable activities dataset consists of 693 videos that contain activities in 16 classes performed by 14 actors. Each activity is composed of 3 to 11 atomic actions. RGB-D data for each sequence is captured using a Microsoft Kinect sensor and estimate position of relevant body joints. The dataset provides annotations of the activity for each video and the actions for each of the four human parts (left/right arm and leg) for each frame in every video. Source: Discriminative Hierarchical Modeling of Spatio-Temporally Composable Human Activities Image Source: https://ialillo.sitios.ing.uc.cl/ActionsCVPR2014/

CORSMAL is a dataset for estimating the position and orientation in 3D (or 6D pose) of an object from a single view. The dataset consists of 138,240 images of rendered hands and forearms holding 48 synthetic objects, split into 3 grasp categories over 30 real backgrounds. Source: A mixed-reality dataset for category-level 6D pose and size estimation of hand-occluded containers Image Source: https://arxiv.org/pdf/2211.10470v1.pdf

DAD-3DHeads dataset consists of 44,898 images collected from various sources (37,840 in the training set, 4,312 in the validation set, and 2,746 in the test set). The dataset is well-balanced over a wide range of poses, face expressions, and occlusions. The attribute labels are a valuable signal for subgroup analysis and for generalisation to in-the-wild deployment conditions.

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

FEAFA+ is a dataset for Facial expression analysis and 3D Facial animation. It includes 150 video sequences from FEAFA and DISFA, with a total of 230,184 frames being manually annotated on floating-point intensity value of 24 redefined AUs using the Expression Quantitative Tool.

FFHQ-Text is a small-scale face image dataset with large-scale facial attributes, designed for text-to-face generation & manipulation, text-guided facial image manipulation, and other vision-related tasks. This dataset is an extension of the NVIDIA Flickr-Faces-HQ Dataset (FFHQ), which is the selected top 760 female FFHQ images that only contain one complete human face.

Largest, first-of-its-kind, in-the-wild, fine-grained workout/exercise posture analysis dataset, covering three different exercises: BackSquat, Barbell Row, and Overhead Press. Seven different types of exercise errors are covered. Unlabeled data is also provided to facilitate self-supervised learning.

The NVIDIA HOPE datasets consist of RGBD images and video sequences with labeled 6-DoF poses for 28 toy grocery objects. The toy grocery objects are readily available for purchase and have ideal size and weight for robotic manipulation. 3D textured meshes for generating synthetic training data are provided. The HOPE-Image dataset shows the objects in 50 scenes from 10 household/office environments, and contains 188 test images taken in 8 environments, with a total of 40 scenes (unique camera and object poses). Up to 5 lighting variations are captured for each scene, including backlighting and angled direct lighting with cast shadows. Scenes are cluttered with varying levels of occlusion. An additional 50 validation images are included from 2 environments in 10 scene arrangements. Within each scene, up to 5 lighting variations are captured with the same camera and object poses. For example, the captures in valid/scene_0000/*.json all depict the same camera pose and arrangement of objects, but each individual capture (0000.json, 0001.json, ...) has a different lighting condition. For this reason, each image should be treated independently for purposes of pose prediction. The most favorable lighting condition for each scene is found in image 0000.json. Images were captured using a RealSense D415 RGBD camera. Systematic errors were observed in the depth values relative to the estimated distance of a calibration grid. To correct for this, depth frames are scaled by a factor of 0.98042517 before registering to RGB. Annotations were made manually using these corrected RGBD frames. NOTE: Only validation set annotations are included. Test annotations are managed by the BOP challenge.

The HOPE-Video dataset contains 10 video sequences (2038 frames) with 5-20 objects on a tabletop scene captured by a robot arm-mounted RealSense D415 RGBD camera. In each sequence, the camera is moved to capture multiple views of a set of objects in the robotic workspace. First COLMAP was applied to refine the camera poses (keyframes at 6~fps) provided by forward kinematics and RGB calibration from RealSense to Baxter's wrist camera. 3D dense point cloud was then generated via CascadeStereo (included for each sequence in 'scene.ply'). Ground truth poses for the HOPE objects models in the world coordinate system were annotated manually using the CascadeStereo point clouds. The following are provided for each frame: *Camera intrinsics/extrinsics *RGB images of 640x480 *Depth images of 640x480 *3D scene reconstruction from CascadeStereo *Object pose annotation in the camera frame Objects consist of a set of 28 toy grocery items selected for compatibility with robot manipulation and widespread availability. Textured models were generated by an EinScan-SE 3D Scanner, units were converted to centimeters, and the centers/rotations of the meshes were aligned to a canonical pose.

HSPACE (Human-SPACE) is a large-scale photo-realistic dataset of animated humans placed in complex synthetic indoor and outdoor environments. For all frames the dataset provides 3d pose and shape ground truth, as well as other rich image annotations including human segmentation, body part localisation semantics, and temporal correspondences.

A synthetic data of videos of human action sequences and the corresponding optical flow.

The ICVL dataset is a hand pose estimation dataset that consists of 330K training frames and 2 testing sequences with each 800 frames. The dataset is collected from 10 different subjects with 16 hand joint annotations for each frame. Source: AWR: Adaptive Weighting Regression for 3D Hand Pose Estimation Image Source: Tang et al.; Latent Regression Forest: Structured Estimation of 3D Hand Poses

Includes 100K depth images under challenging scenarios. Source: Human Pose Estimation from Depth Images via Inference Embedded Multi-task Learning

Mirrored-Human is a dataset for 3D pose estimation from a single view. It covers a large variety of human subjects, poses and backgrounds. The images are collected from the internet and consists of people in front of mirrors, were both the person and the reflected image are visible. Actions cover dancing, fitness, mirror installation, swing practice

Throughout the history of art, the pose—as the holistic abstraction of the human body's expression—has proven to be a constant in numerous studies. However, due to the enormous amount of data that so far had to be processed by hand, its crucial role to the formulaic recapitulation of art-historical motifs since antiquity could only be highlighted selectively. This is true even for the now automated estimation of human poses, as domain-specific, sufficiently large data sets required for training computational models are either not publicly available or not indexed at a fine enough granularity. With the Poses of People in Art data set, we introduce the first openly licensed data set for estimating human poses in art and validating human pose estimators. It consists of 2,454 images from 22 art-historical depiction styles, including those that have increasingly turned away from lifelike representations of the body since the 19th century. A total of 10,749 human figures are precisely enclosed by rectangular bounding boxes, with a maximum of four per image labeled by up to 17 keypoints; among these are mainly joints such as elbows and knees. For machine learning purposes, the data set is divided into three subsets—training, validation, and testing—, that follow the established JSON-based Microsoft COCO format, respectively. Each image annotation, in addition to mandatory fields, provides metadata from the art-historical online encyclopedia WikiArt.

QST contains 1,167 video clips that are cut out from 216 time-lapse 4K videos collected from YouTube, which can be used for a variety of tasks, such as (high-resolution) video generation, (high-resolution) video prediction, (high-resolution) image generation, texture generation, image inpainting, image/video super-resolution, image/video colorization, image/video animating, etc. Each short clip contains multiple frames (from a minimum of 58 frames to a maximum of 1,200 frames, a total of 285,446 frames), and the resolution of each frame is more than 1,024 x 1,024. Specifically, QST consists of a training set (containing 1000 clips, totally 244,930 frames), a validation set (containing 100 clips, totally 23,200 frames), and a testing set (containing 67 clips, totally 17,316 frames). Click here (Key: qst1) to download the QST dataset.