3D Pose Estimation

The Human3.6M dataset is one of the largest motion capture datasets, which consists of 3.6 million human poses and corresponding images captured by a high-speed motion capture system. There are 4 high-resolution progressive scan cameras to acquire video data at 50 Hz. The dataset contains activities by 11 professional actors in 17 scenarios: discussion, smoking, taking photo, talking on the phone, etc., as well as provides accurate 3D joint positions and high-resolution videos. Source: Space-Time Representation of People Based on 3D Skeletal Data: A Review Image Source: Yu et al

The Leeds Sports Pose (LSP) dataset is widely used as the benchmark for human pose estimation. The original LSP dataset contains 2,000 images of sportspersons gathered from Flickr, 1000 for training and 1000 for testing. Each image is annotated with 14 joint locations, where left and right joints are consistently labelled from a person-centric viewpoint. The extended LSP dataset contains additional 10,000 images labeled for training. Source: Deep Deformation Network for Object Landmark Localization Image: Sumer et al

For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. Hypersim is a photorealistic synthetic dataset for holistic indoor scene understanding. It contains 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry. Source: https://github.com/apple/ml-hypersim Image Source: https://github.com/apple/ml-hypersim

Dataset produced for the SAPIEN simulation environment. From the website: "PartNet-Mobility dataset is a collection of 2K articulated objects with motion annotations and rendernig material. The dataset powers research for generalizable computer vision and manipulation. The dataset is a continuation of ShapeNet and PartNet. "

A hand-object interaction dataset with 3D pose annotations of hand and object. The dataset contains 66,034 training images and 11,524 test images from a total of 68 sequences. The sequences are captured in multi-camera and single-camera setups and contain 10 different subjects manipulating 10 different objects from YCB dataset. The annotations are automatically obtained using an optimization algorithm. The hand pose annotations for the test set are withheld and the accuracy of the algorithms on the test set can be evaluated with standard metrics using the CodaLab challenge submission(see project page). The object pose annotations for the test and train set are provided along with the dataset.

MuCo-3DHP is a large scale training data set showing real images of sophisticated multi-person interactions and occlusions. Source: Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB

ApolloCar3DT is a dataset that contains 5,277 driving images and over 60K car instances, where each car is fitted with an industry-grade 3D CAD model with absolute model size and semantically labelled keypoints. This dataset is above 20 times larger than PASCAL3D+ and KITTI, the current state-of-the-art. Source: ApolloCar3D: A Large 3D Car Instance Understanding Benchmark for Autonomous Driving Image Source: http://apolloscape.auto/car_instance.html

Contains bounding boxes for object detection, instance segmentation masks, keypoints for pose estimation and depth images for each synthetic scenery as well as images for each individual seat for classification. Source: SVIRO: Synthetic Vehicle Interior Rear Seat Occupancy Dataset and Benchmark

Multi-View Operating Room (MVOR) is a dataset recorded during real clinical interventions. It consists of 732 synchronized multi-view frames recorded by three RGB-D cameras in a hybrid OR. It also includes the visual challenges present in such environments, such as occlusions and clutter. Source: MVOR: A Multi-view RGB-D Operating Room Dataset for 2D and 3D Human Pose Estimation Source: https://github.com/CAMMA-public/mvor Image Source: https://github.com/CAMMA-public/mvor

UnrealEgo is a dataset that provides in-the-wild stereo images with a large variety of motions for 3D human pose estimation. The in-the-wild stereo images are stereo fisheye images and depth maps with a resolution of 1024×1024 pixels each with 25 frames per second and a total of 450k (900k images) are captured for the dataset. Metadata is provided for each frame, including 3D joint positions, camera positions, and 2D coordinates of reprojected joint positions in the fisheye views.

Unite The People is a dataset for 3D body estimation. The images come from the Leeds Sports Pose dataset and its extended version, as well as the single person tagged people from the MPII Human Pose Dataset. The images are labeled with different types of annotations such as segmentation labels, pose or 3D.

We provide manual annotations of 14 semantic keypoints for 100,000 car instances (sedan, suv, bus, and truck) from 53,000 images captured from 18 moving cameras at Multiple intersections in Pittsburgh, PA. Please fill the google form to get a email with the download links:

HUMAN4D is a large and multimodal 4D dataset that contains a variety of human activities simultaneously captured by a professional marker-based MoCap, a volumetric capture and an audio recording system. By capturing 2 female and $2$ male professional actors performing various full-body movements and expressions, HUMAN4D provides a diverse set of motions and poses encountered as part of single- and multi-person daily, physical and social activities (jumping, dancing, etc. ), along with multi-RGBD (mRGBD), volumetric and audio data. Description from: HUMAN4D: A Human-Centric Multimodal Dataset for Motions and Immersive Media Image source: https://github.com/tofis/human4d_dataset

multi-view imagery of people interacting with a variety of rich 3D environments

Enhancing the robustness of vision algorithms in real-world scenarios is challenging. One reason is that existing robustness benchmarks are limited, as they either rely on synthetic data or ignore the effects of individual nuisance factors. We introduce OOD-CV, a benchmark dataset that includes out-of-distribution examples of 10 object categories in terms of pose, shape, texture, context and the weather conditions, and enables benchmarking models for image classification, object detection, and 3D pose estimation. In addition to this novel dataset, we contribute extensive experiments using popular baseline methods, which reveal that: 1. Some nuisance factors have a much stronger negative effect on the performance compared to others, also depending on the vision task. 2. Current approaches to enhance robustness have only marginal effects, and can even reduce robustness. 3. We do not observe significant differences between convolutional and transformer architectures. We believe our dataset provides a rich test bed to study robustness and will help push forward research in this area.

We introduce HARPER, a novel dataset for 3D body pose estimation and forecast in dyadic interactions between users and \spot, the quadruped robot manufactured by Boston Dynamics. The key-novelty is the focus on the robot's perspective, i.e., on the data captured by the robot's sensors. These make 3D body pose analysis challenging because being close to the ground captures humans only partially. The scenario underlying HARPER includes 15 actions, of which 10 involve physical contact between the robot and users. The Corpus contains not only the recordings of the built-in stereo cameras of Spot, but also those of a 6-camera OptiTrack system (all recordings are synchronized). This leads to ground-truth skeletal representations with a precision lower than a millimeter. In addition, the Corpus includes reproducible benchmarks on 3D Human Pose Estimation, Human Pose Forecasting, and Collision Prediction, all based on publicly available baseline approaches. This enables future HARPER users to rigorously compare their results with those we provide in this work. Source: [Download dataset] (https://github.com/intelligolabs/HARPER)

PedX is a large-scale multi-modal collection of pedestrians at complex urban intersections. The dataset provides high-resolution stereo images and LiDAR data with manual 2D and automatic 3D annotations. The data was captured using two pairs of stereo cameras and four Velodyne LiDAR sensors. Source: pedx.io

Accurate 3D human pose estimation is essential for sports analytics, coaching, and injury prevention. However, existing datasets for monocular pose estimation do not adequately capture the challenging and dynamic nature of sports movements. In response, we introduce SportsPose, a large-scale 3D human pose dataset consisting of highly dynamic sports movements. With more than 176,000 3D poses from 24 different subjects performing 5 different sports activities, SportsPose provides a diverse and comprehensive set of 3D poses that reflect the complex and dynamic nature of sports movements. Contrary to other markerless datasets we have quantitatively evaluated the precision of SportsPose by comparing our poses with a commercial marker-based system and achieve a mean error of 34.5 mm across all evaluation sequences. This is comparable to the error reported on the commonly used 3DPW dataset. We further introduce a new metric, local movement, which describes the movement of the wrist and ankle joints in relation to the body. With this, we show that SportsPose contains more movement than the Human3.6M and 3DPW datasets in these extremum joints, indicating that our movements are more dynamic. The dataset with accompanying code can be downloaded from our website. We hope that SportsPose will allow researchers and practitioners to develop and evaluate more effective models for the analysis of sports performance and injury prevention. With its realistic and diverse dataset, SportsPose provides a valuable resource for advancing the state-of-the-art in pose estimation in sports

The dataset is designed specifically to solve a range of computer vision problems (2D-3D tracking, posture) faced by biologists while designing behavior studies with animals. Typically, datasets for animal-specific vision tasks are created using open-source video material. This might be effective for an initial start, but these methods are not deployment ready for the behavior community. Therefore, we designed a semi-automated method for biologists to create well-curated datasets at a large scale for the ML and Vision community. 3D-POP is the first dataset with 3D ground truth for multi-animal, multi-view tracking problems. Highlight: The dataset is captured with the intention of using it for various vision problems and with different levels of complexity (no of cameras, no of individuals) Video explanation: Link to YouTube video Video teaser: Link to YouTube video ## Dataset Features: ### Marker-based videos: - 6 hours+ of annotations of 18 individuals (groups of 1, 2, 5, 10). - Bounding box - Trajectories (2D and 3D) - Posture (2D and 3D) with 9 key points - Identities - Total of 57 sequences (4K) with 4 views. - Dataset customization* (Users can modify the dataset and add key points to the dataset) ### Markerless: - 1Hr+ videos of 18 individuals in groups of 1, 2, 5, 11. The birds have no markers. This data is provided as test cases and unsupervised approaches. ## Problems: ### 2D domain: - Position, Posture of birds (different group sizes n = 1, 2, 5, 10) with Single/Multiview. - Tracking with single - multiview ### 3D domain: - Position, Posture of birds (different group sizes n = 1, 2, 5, 10) with Single/Multiview. - Tracking with single - multiview ### Fine-grained recognition: - Identity tracking with ground truth. ### Unsupervised learning: - 2D or 3D posture problems ## Idea: The dataset is created with a motion capture system, using the 6-DOF tracking ability. Assumptions are that head and body act as rigid bodies when birds walk and forage (proved with experiment). Therefore, we get the 3D position of key points by tracking head/body orientation.

Largest, first-of-its-kind, in-the-wild, fine-grained workout/exercise posture analysis dataset, covering three different exercises: BackSquat, Barbell Row, and Overhead Press. Seven different types of exercise errors are covered. Unlabeled data is also provided to facilitate self-supervised learning.

HSPACE (Human-SPACE) is a large-scale photo-realistic dataset of animated humans placed in complex synthetic indoor and outdoor environments. For all frames the dataset provides 3d pose and shape ground truth, as well as other rich image annotations including human segmentation, body part localisation semantics, and temporal correspondences.

Includes 100K depth images under challenging scenarios. Source: Human Pose Estimation from Depth Images via Inference Embedded Multi-task Learning

Mirrored-Human is a dataset for 3D pose estimation from a single view. It covers a large variety of human subjects, poses and backgrounds. The images are collected from the internet and consists of people in front of mirrors, were both the person and the reflected image are visible. Actions cover dancing, fitness, mirror installation, swing practice

This dataset presents a vision and perception research dataset collected in Rome, featuring RGB data, 3D point clouds, IMU, and GPS data. We introduce a new benchmark targeting visual odometry and SLAM, to advance the research in autonomous robotics and computer vision. This work complements existing datasets by simultaneously addressing several issues, such as environment diversity, motion patterns, and sensor frequency. It uses up-to-date devices and presents effective procedures to accurately calibrate the intrinsic and extrinsic of the sensors while addressing temporal synchronization. During recording, we cover multi-floor buildings, gardens, urban and highway scenarios. Combining handheld and car-based data collections, our setup can simulate any robot (quadrupeds, quadrotors, autonomous vehicles). The dataset includes an accurate 6-dof ground truth based on a novel methodology that refines the RTK-GPS estimate with LiDAR point clouds through Bundle Adjustment. All sequences divided in training and testing are accessible through our website.

DRACO20K dataset is used for evaluating object canonicalization on methods that estimate a canonical frame from a monocular input image. Provides: 1. Mixed Reality Multi-view RGB-D images rendered from ShapeNet objects 2. Camera poses 3. NOCS maps 4. Semantic 2D keypoints with visibility 5. Object-centric mask

Estimating camera motion in deformable scenes poses a complex and open research challenge. Most existing non-rigid structure from motion techniques assume to observe also static scene parts besides deforming scene parts in order to establish an anchoring reference. However, this assumption does not hold true in certain relevant application cases such as endoscopies. To tackle this issue with a common benchmark, we introduce the Drunkard’s Dataset, a challenging collection of synthetic data targeting visual navigation and reconstruction in deformable environments. This dataset is the first large set of exploratory camera trajectories with ground truth inside 3D scenes where every surface exhibits non-rigid deformations over time. Simulations in realistic 3D buildings lets us obtain a vast amount of data and ground truth labels, including camera poses, RGB images and depth, optical flow and normal maps at high resolution and quality.

Event-Human3.6m is a challenging dataset for event-based human pose estimation by simulating events from the RGB Human3.6m dataset. It is built by converting the RGB recordings of Human3.6m into events and synchronising raw joints ground-truth with events frames through interpolation.

A Simulated Benchmark for multi-modal SLAM Systems Evaluation in Large-scale Dynamic Environments.

This dataset comprehends the 3D building information model (in IFC and Revit formats), manually elaborated based on the terrestrial laser scanner of the sequence 2 of ConSLAM, and the refined ground truth (GT) poses (in TUM format) of sessions 2, 3, 4, and 5 of the open-access ConSLAM dataset (which provides camera, LiDAR, and IMU measurements). This dataset enables the evaluation of pose estimation algorithms, such as localization, odometry, SLAM, and Bundle adjustment. Given the differences between the real-world and the 3D reference BIM model (Scan-Map deviations), the dataset is appropriate for evaluating long-term localization and mapping capabilities, i.e., methods that are robust to changes in the environment. In particular, in a real-world construction environment with normal clutter (not as in the Hilti dataset).

Accidental Turntables contains a challenging set of 41,212 images of cars in cluttered backgrounds, motion blur and illumination changes that serves as a benchmark for 3D pose estimation. Source: Accidental Turntables: Learning 3D Pose by Watching Objects Turn Image Source: https://people.cs.umass.edu/~zezhoucheng/acci-turn/

ConSLAM is a real-world dataset collected periodically on a construction site to measure the accuracy of mobile scanners' SLAM algorithms. The dataset contains time-synchronized and spatially registered RGB and NIR images and 360-deg LiDAR scans, 9-axis IMU measurements, and professional ground-truth terrestrial laser scans. This dataset reflects the periodic need to scan construction sites with the aim of accurately monitoring progress using a hand-held scanner. The sensors used for data acquisition are: - LiDAR: Velodyne VLP-16. - RGB camera: Alvium U-319c, 3.2 MP. - NIR camera: Alvium 1800 U-501, 5.0 MP. - 9-axis IMU: Xsens MTi-610.

A new large-scale dataset that consists of 409 fine-grained categories and 31,881 images with accurate 3D pose annotation. Source: Improving Annotation for 3D Pose Dataset of Fine-Grained Object Categories

Human pose estimation (HPE) in the top-view using fisheye cameras presents a promising and innovative application domain. However, the availability of datasets capturing this viewpoint is extremely limited, especially those with high-quality 2D and 3D keypoint annotations. Addressing this gap, we leverage the capabilities of Neural Radiance Fields (NeRF) technique to establish a comprehensive pipeline for generating human pose datasets from existing 2D and 3D datasets, specifically tailored for the top-view fisheye perspective. Through this pipeline, we create a novel dataset NToP (NeRF-powered Top-view human Pose dataset for fisheye cameras) with over 570 thousand images, and conduct an extensive evaluation of its efficacy in enhancing neural networks for 2D and 3D top-view human pose estimation. Extensive evaluations on existing top-view 2D and 3D HPE datasets as well as our new real-world top-view 2D HPE dataset OmniLab prove that our dataset is effective and exceeds previous datasets in this field of research.

The Store Dataset is a dataset for estimating 3D poses of multiple humans in real-time. It is captured inside two kinds of simulated stores with 12 and 28 cameras, respectively. Source: https://arxiv.org/abs/2003.03972

This is a pose estimation dataset, consisting of symmetric 3D shapes where multiple orientations are visually indistinguishable. The challenge is to predict all equivalent orientations when only one orientation is paired with each image during training (as is the scenario for most pose estimation datasets). In contrast to most pose estimation datasets, the full set of equivalent orientations is available for evaluation. There are eight shapes total, each rendered from 50,000 viewpoints distributed uniformly at random over the full space of 3D rotations. Five of the shapes are featureless -- tetrahedron, cube, icosahedron, cone, and cylinder. Of those, the three Platonic solids (tetrahedron, cube, icosahedron) are annotated with their 12, 24, and 60 discrete symmetries, respectively. The cone and cylinder are annotated with their continuous symmetries discretized at 1 degree intervals. These symmetries are provided for evaluation; the intended supervision is only a single rotation with each image. The remaining three shapes are marked with a distinguishing feature. There is a tetrahedron with one red-colored face, a cylinder with an off-center dot, and a sphere with an X capped by a dot. Whether or not the distinguishing feature is visible, the space of possible orientations is reduced. We do not provide the set of equivalent rotations for these shapes. Each example contains of - the 224x224 RGB image - a shape index so that the dataset may be filtered by shape. The indices correspond to: 0 = tetrahedron 1 = cube 2 = icosahedron 3 = cone 4 = cylinder 5 = marked tetrahedron 6 = marked cylinder 7 = marked sphere - the rotation used in the rendering process, represented as a 3x3 rotation matrix - the set of known equivalent rotations under symmetry, for evaluation. In the case of the three marked shapes, this is only the rendering rotation.

InfiniteRep is a synthetic, open-source dataset for fitness and physical therapy (PT) applications. It includes 1k videos of diverse avatars performing multiple repetitions of common exercises. It includes significant variation in the environment, lighting conditions, avatar demographics, and movement trajectories. From cadence to kinematic trajectory, each rep is done slightly differently -- just like real humans. InfiniteRep videos are accompanied by a rich set of pixel-perfect labels and annotations, including frame-specific repetition counts. The dataset features: + 100 videos per exercise, spanning 5 to 10 repetitions each (1,000 videos total) + 7 unique indoor scenes + Realistic environmental occlusion (+ corresponding labels) + Diverse lighting conditions + Varied body shape, skin tones, and clothing + Rich annotations for 2D and 3D supervision ## Exercises The dataset currently includes the following exercises: + Pushups + Alternating Bicep Curls (with dumbbells) + Delt Flys (with dumbbells) + Squats + Bird Dogs + Supermans + Bicycle Crunches + Leg Raises + Front Raises (with dumbbells) + Overhead Press (with dumbbells) ## Annotations The dataset includes the following annotations: + Bounding boxes + Segmentation masks + Keypoints + Joint angles (quaternions) + Percent occlusion + Avatar characteristics + Camera position + and more Want depth labels? They are not included in the dataset but we can send them to you. Email us at info@toinfinity.ai. ## Download Download the dataset: toinfinity.ai/infiniterep Github repo with additional documentation: https://github.com/toinfinityai/InfiniteRep ## Need more data? Infinity AI specializes in generating custom synthetic data. If you need more (or different data), drop us a line at info@toinfinity.ai (we read every email).