6D Pose Estimation

The YCB-Video dataset is a large-scale video dataset for 6D object pose estimation. provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames. Source: https://rse-lab.cs.washington.edu/projects/posecnn/ Image Source: https://www.researchgate.net/figure/Examples-of-refined-poses-on-the-YCB-Video-dataset-which-use-results-from-PoseCNN-Xiang_fig6_339663565

DexYCB is a dataset for capturing hand grasping of objects. It can be used three relevant tasks: 2D object and keypoint detection, 6D object pose estimation, and 3D hand pose estimation. The dataset was built using 20 objects from the YCB-Video dataset, and consists of multiple trials from 10 subjects. For each trial, there is a target object with 2 to 4 other objects placed on a table. The subject is asked to start from a relaxed pose, pick up the target object, and hold it in the air. Some subjects were asked to pretend to hand over the object to someone across from them. Each action is recorded for 3 seconds, repeating the trial 5 times for each target object, each time with a random set of accompanied objects and placement. In total there are 100 trials per subject, and 1,000 trials in total for all subjects. Source: DexYCB: A Benchmark for Capturing Hand Grasping of Objects

REAL275 is a benchmark for category-level pose estimation. It contains 4300 training frames, 950 validation and 2750 for testing across 18 different real scenes.

A hand-object interaction dataset with 3D pose annotations of hand and object. The dataset contains 66,034 training images and 11,524 test images from a total of 68 sequences. The sequences are captured in multi-camera and single-camera setups and contain 10 different subjects manipulating 10 different objects from YCB dataset. The annotations are automatically obtained using an optimization algorithm. The hand pose annotations for the test set are withheld and the accuracy of the algorithms on the test set can be evaluated with standard metrics using the CodaLab challenge submission(see project page). The object pose annotations for the test and train set are provided along with the dataset.

The LM (Linemod) dataset is a valuable resource introduced by Stefan Hinterstoisser and colleagues in their research on model-based training, detection, and pose estimation of texture-less 3D objects in heavily cluttered scenes¹. Let's delve into the details: 1. Purpose and Context: - The primary goal of the LM dataset is to facilitate the development and evaluation of methods for detecting and estimating the 6 degrees-of-freedom pose of texture-less 3D objects. - It specifically targets scenarios where objects lack distinctive textures and are embedded in complex backgrounds with occlusions. 2. Methodology: - The dataset builds upon the LINEMOD approach, which combines depth and color information to create templates representing different views of an object. - LINEMOD templates are learned from 3D models and serve as a basis for object detection. - The initial LINEMOD method had limitations, including online template learning and approximate pose estimation. 3. Improvements and Contributions: - Hinterstoisser et al. enhance LINEMOD by incorporating accurate 3D models of objects. - Their approach leverages the 3D model to address the shortcomings of the original LINEMOD. - Notable improvements include better pose estimation and reduced false positives. - The proposed framework is suitable for robotics applications, such as object manipulation. 4. Dataset Details: - The LM dataset consists of 15 registered video sequences, each containing over 1100 frames. - These sequences feature 15 different texture-less household objects. - Objects in the dataset exhibit discriminative color, shape, and size characteristics. - Researchers can use this dataset to evaluate and compare their methods for object detection and pose estimation. LM Dataset Example In summary, the LM dataset provides a valuable benchmark for advancing the field of 6D object pose estimation, especially in scenarios where texture information is limited¹². Researchers can access this dataset to test and refine their algorithms, ultimately contributing to advancements in robotics and machine vision. (1) Model Based Training, Detection and Pose ... - Stefan HINTERSTOISSER. http://stefan-hinterstoisser.com/papers/hinterstoisser2012accv.pdf. (2) Datasets - BOP: Benchmark for 6D Object Pose Estimation. https://bop.felk.cvut.cz/datasets/. (3) paroj/linemod_dataset: Hinterstoisser et al. ACCV12 dataset - GitHub. https://github.com/paroj/linemod_dataset.

HomebrewedDB is a dataset for 6D pose estimation mainly targeting training from 3D models (both textured and textureless), scalability, occlusions, and changes in light conditions and object appearance. The dataset features 33 objects (17 toy, 8 household and 8 industry-relevant objects) over 13 scenes of various difficulty. It also consists of a set of benchmarks to test various desired detector properties, particularly focusing on scalability with respect to the number of objects and resistance to changing light conditions, occlusions and clutter.

ARCTIC is a dataset of free-form interactions of hands and articulated objects. ARCTIC has 1.2M images paired with accurate 3D meshes for both hands and for objects that move and deform over time. The dataset also provides hand-object contact information.

ApolloCar3DT is a dataset that contains 5,277 driving images and over 60K car instances, where each car is fitted with an industry-grade 3D CAD model with absolute model size and semantically labelled keypoints. This dataset is above 20 times larger than PASCAL3D+ and KITTI, the current state-of-the-art. Source: ApolloCar3D: A Large 3D Car Instance Understanding Benchmark for Autonomous Driving Image Source: http://apolloscape.auto/car_instance.html

Are current 3D object tracking methods truely robust enough for low-fidelity depth sensors like the iPhone LiDAR? We introduce DTTD-Mobile (fully compatible w/ YCB toolbox), a new benchmark built on real-world data captured from mobile devices; 18 objects observed in 100 videos with 47,668 sampled frames and 114,143 object annotations. We evaluate several popular methods—including BundleSDF, ES6D, MegaPose, and DenseFusion—and highlight their limitations in this challenging setting. Keywords: ObjectPoseEstimation, MobileAI, EdgeAI, ARVR, CVPR

A new dataset with significant occlusions related to object manipulation. Source: se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains

The DREAM dataset is introduce by the paper "Camera-to-Robot Pose Estimation from a Single Image" (ICRA 2020). This dataset consists of synthetic images (both with and without domain randomlization) of three different robot manipulators (Franka Emika’s Panda, Kuka’s LBR iiwa 7 R800, and Rethink Robotics’ Baxter) , as well as real-world images of Franka Emika’s Panda taken from various RGBD cameras (XBox 360 Kinect (XK), RealSense (RS), and Azure Kinect (AK)). Each instance in the dataset contains an RGB image, keypoint 3D/2D coordinates , global camera-to-robot transformation and joint state configurations (from both revolute and prismatic joint) of the robot. Tasks like estimating robot pose (camera pose) from a single RGB image, camera-to-robot calibration can be conducted and evaluated in this dataset.

We present a large-scale dataset for 3D urban scene understanding. Compared to existing datasets, our dataset consists of 75 outdoor urban scenes with diverse backgrounds, encompassing over 15,000 images. These scenes offer 360◦ hemispherical views, capturing diverse foreground objects illuminated under various lighting conditions. Additionally, our dataset encompasses scenes that are not limited to forward-driving views, addressing the limitations of previous datasets such as limited overlap and coverage between camera views. The closest pre-existing dataset for generalizable evaluation is DTU [2] (80 scenes) which comprises mostly indoor objects and does not provide multiple foreground objects or background scenes. We use the Parallel Domain synthetic data generation to render high-fidelity 360◦ scenes. We select 3 different maps i.e. SF 6thAndMission, SF GrantAndCalifornia and SF VanNessAveAndTurkSt and sample 75 different scenes across all 3 maps as our backgrounds (All 75 scenes across 3 maps are significantly different road scenes from each other, captured at different viewpoints in the city). We select 20 different cars in 50 different textures for training and randomly sample from 1 to 4 cars to render in a scene. We refer to this dataset as NeRDS 360: NeRF for Reconstruction, Decomposition and Scene Synthesis of 360◦ outdoor scenes. In total, we generate 15k renderings by sampling 200 cameras in a hemispherical dome at a fixed distance from the center of cars. We held out 5 scenes with 4 different cars and different backgrounds for testing, comprising 100 cameras distributed uniformly sampled in the upper hemisphere, different from the camera distributions used for training. We use the diverse validation camera distribution to test our approach’s ability to generalize to unseen viewpoints as well as unseen scenes during training. Our dataset and the corresponding task is extremely challenging due to occlusions, diversity of backgrounds, and rendered objects with various lightning and shadows. Our task entails reconstructing 360◦ hemispherical views of complete scenes using a handful of observations i.e. 1 to 5 as shown by red cameras whereas evaluating using all 100 hemispherical views, hence our task requires strong priors for novel view synthesis of outdoor scenes. Tasks our dataset support: - Generaliazable Novel view synthesis (Few shot evaluation) - Novel view synthesis (Overfitting evaluation) - 6D pose estimation - Object editing - Depth estimation - Semantic Segmentation - Instance Segmentation

CORSMAL is a dataset for estimating the position and orientation in 3D (or 6D pose) of an object from a single view. The dataset consists of 138,240 images of rendered hands and forearms holding 48 synthetic objects, split into 3 grasp categories over 30 real backgrounds. Source: A mixed-reality dataset for category-level 6D pose and size estimation of hand-occluded containers Image Source: https://arxiv.org/pdf/2211.10470v1.pdf

Estimating camera motion in deformable scenes poses a complex and open research challenge. Most existing non-rigid structure from motion techniques assume to observe also static scene parts besides deforming scene parts in order to establish an anchoring reference. However, this assumption does not hold true in certain relevant application cases such as endoscopies. To tackle this issue with a common benchmark, we introduce the Drunkard’s Dataset, a challenging collection of synthetic data targeting visual navigation and reconstruction in deformable environments. This dataset is the first large set of exploratory camera trajectories with ground truth inside 3D scenes where every surface exhibits non-rigid deformations over time. Simulations in realistic 3D buildings lets us obtain a vast amount of data and ground truth labels, including camera poses, RGB images and depth, optical flow and normal maps at high resolution and quality.

Accurately tracking the six degree-of-freedom pose of an object in real scenes is an important task in computer vision and augmented reality with numerous applications. Although a variety of algorithms for this task have been proposed, it remains difficult to evaluate existing methods in the literature as oftentimes different sequences are used and no large benchmark datasets close to real-world scenarios are available. In this paper, we present a large object pose tracking benchmark dataset consisting of RGB-D video sequences of 2D and 3D targets with ground-truth information. The videos are recorded under various lighting conditions, different motion patterns and speeds with the help of a programmable robotic arm. We present extensive quantitative evaluation results of the state-of-the-art methods on this benchmark dataset and discuss the potential research directions in this field.

This dataset comprehends the 3D building information model (in IFC and Revit formats), manually elaborated based on the terrestrial laser scanner of the sequence 2 of ConSLAM, and the refined ground truth (GT) poses (in TUM format) of sessions 2, 3, 4, and 5 of the open-access ConSLAM dataset (which provides camera, LiDAR, and IMU measurements). This dataset enables the evaluation of pose estimation algorithms, such as localization, odometry, SLAM, and Bundle adjustment. Given the differences between the real-world and the 3D reference BIM model (Scan-Map deviations), the dataset is appropriate for evaluating long-term localization and mapping capabilities, i.e., methods that are robust to changes in the environment. In particular, in a real-world construction environment with normal clutter (not as in the Hilti dataset).

UW Indoor Scenes (UW-IS) Occluded dataset is curated using commodity hardware (Intel RealSense D435) to reflect real world robotics scenarios. It consists of two completely different indoor environments. The first environment is a lounge where the objects are placed on a tabletop. The second environment is a mock warehouse setup where the objects are placed on a shelf. For each of these environments, we have RGB-D images from 36 videos comprising five to seven objects each, taken from distances up to approximately 2m. The videos cover two different lighting conditions, three different levels of object separation for three different object categories (i.e., kitchen objects, food items, and tools/miscellaneous). The first level of object separation is such that there is no object occlusion. The second level of object separation is such that some occlusion occurs, while the third level is where the objects are placed extremely close together. Overall, the dataset considers 20 object classes and consists of 8,456 images, which have a total of 42,902 object instances. We also provide instance segmentation masks and 6D pose annotations for all the images generated using LabelFusion (Marion et al., 2018)

Data used for the paper SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data It contains over 1GB of high-quality motion capture data recorded with an Xsens Awinda system while using a variety of VR applications in Meta Quest devices.

Dataset consist of both real captures from Photoneo PhoXi structured light scanner devices annotated by hand and synthetic samples produced by custom generator. In comparison with existing datasets for 6D pose estimation, some notable differences include: * most of the captured bins are texture-less, made from uniform, single-colored materials, * all bins are of cuboid shape with different proportions. Compared to objects with complex geometry, bins consist of flat faces with edges, which are not guaranteed to be seen in the capture due to occlusion. Surface models of these bins are not provided, just their approximate bounding boxes, * PhoXi scanner provides high-resolution 3D geometry data, but no RGB data, with a rough and noisy gray-scale intensity image being the closest equivalent, * captures come from different devices with various intrinsic camera parameters. 3D point clouds contain these parameters implicitly as opposed to RGBD images. Due to its currently limited size, we recommend cross-validation instead of an explicit train-validation split. We plan to add more samples into the dataset. NOTE: Annotation files with suffixes _bad, _ish or _catastrophic should be ignored. It was not possible to annotate them correctly with our current toolset.

ConSLAM is a real-world dataset collected periodically on a construction site to measure the accuracy of mobile scanners' SLAM algorithms. The dataset contains time-synchronized and spatially registered RGB and NIR images and 360-deg LiDAR scans, 9-axis IMU measurements, and professional ground-truth terrestrial laser scans. This dataset reflects the periodic need to scan construction sites with the aim of accurately monitoring progress using a hand-held scanner. The sensors used for data acquisition are: - LiDAR: Velodyne VLP-16. - RGB camera: Alvium U-319c, 3.2 MP. - NIR camera: Alvium 1800 U-501, 5.0 MP. - 9-axis IMU: Xsens MTi-610.

GraspClutter6D is a large-scale real-world dataset for robust object perception and robotic grasping in cluttered environments. It features 1,000 highly cluttered scenes with dense arrangements (average 14.1 objects/scene with 62.6% occlusion), 200 household, industrial, and warehouse objects captured in 75 diverse environment configurations (bins, shelves, and tables), multi-view data from 4 RGB-D cameras (RealSense D415, D435, Azure Kinect, and Zivid One+), and comprehensive annotations including 736K 6D object poses and 9.3 billion feasible robotic grasps for 52K RGB-D images. The dataset provides a challenging testbed for segmentation, 6D pose estimation, and grasp detection algorithms in realistic cluttered scenarios.

This data set contains over 600GB of multimodal data from a Mars analog mission, including accurate 6DoF outdoor ground truth, indoor-outdoor transitions with continuous cross-domain ground truth, and indoor data with Optitrack measurements as ground truth. With 26 flights and a combined distance of 2.5km, this data set provides you with various distinct challenges for testing and proofing your algorithms. The UAV carries 18 sensors, including a high-resolution navigation camera and a stereo camera with an overlapping field of view, two RTK GNSS sensors with centimeter accuracy, as well as three IMUs, placed at strategic locations: Hardware dampened at the center, off-center with a lever arm, and a 1kHz IMU rigidly attached to the UAV (in case you want to work with unfiltered data). The sensors are fully pre-calibrated, and the data set is ready to use. However, if you want to use your own calibration algorithms, then the raw calibration data is also ready for download. The cross-domain outdoor to indoor transition segments are especially challenging because of realistic sensor behavior such as GNSS degradation and dropouts, changes in the measured magnetic field, and changing light conditions when transitioning to the indoor environment. The data set also provides 4 hours of static sensor data and vibration data with accurate RPM measurements to provide you with additional information about sensor properties and vehicle integrity. The data set provides you with everything you need to test your research: First, you can test your algorithms with the indoor data sets in a controlled environment and then switch to the more challenging flight scenario, such as the transition data, which requires sensor switching, or the Mars analog data with higher velocities, multiple touchdowns, challenging ground structures or constant velocity segments. The Mars analog data also contains cliff flight overs and traverses while the stereo camera faces the cliff in case you want to perform a 3D reconstruction or challenge your SLAM algorithm. The critical aspects of each data set are shown on the website (https://sst.aau.at/cns/datasets), making it easy to find the best data to test or challenge your algorithm.

KITTI-6DoF is a dataset that contains annotations for the 6DoF estimation task for 5 object categories on 7,481 frames. Source: Simultaneous Multiple Object Detection and Pose Estimation using 3D Model Infusion with Monocular Vision Image Source: https://arxiv.org/pdf/2211.11188v1.pdf

The SMOT dataset, Single sequence-Multi Objects Training, is collected to represent a practical scenario of collecting training images of new objects in the real world, i.e. a mobile robot with an RGB-D camera collects a sequence of frames while driving around a table to learning multiple objects and tries to recognize objects in different locations. Source: SMOT

Data and experiments for motion-based extrinsic calibration using [trajectory_calibration]. The data was generated for evaluating hand-eye calibration algorithms in extrinsic sensor-to-sensor calibration. The repository is organised as follows: ├── calib │ └── *.csv # The output calibration values ├── kitti_tests │ └── ... # The trajectories generated from KITTI data ├── matlab │ └── ... # Matlab files to generate simulation data ├── simulation_tests │ └── ... # The generated simulation data ├── *.py # Scripts to run the calibration experiments └── README.md If you use the data in an academic context, please cite: @article{valimaki2023, author = {Välimäki, Tuomas and Garigipati, Bharath and Ghabcheloo, Reza}, title = {Motion-Based Extrinsic Sensor-to-Sensor Calibration: Effect of Reference Frame Selection for New and Existing Methods}, journal = {Sensors}, volume = {23}, year = {2023}, number = {7}, pages = {3740}, url = {https://www.mdpi.com/1424-8220/23/7/3740}, issn = {1424-8220}, doi = {10.3390/s23073740} } [trajectory_calibration]: https://github.com/tau-alma/trajectory_calibration

The YCB-Ev dataset contains synchronized RGB-D frames and event data that enables evaluating 6DoF object pose estimation algorithms using these modalities. This dataset provides ground truth 6DoF object poses for the same 21 YCB objects that were used in the YCB-Video (YCB-V) dataset, allowing for cross-dataset algorithm performance evaluation. The dataset consists of 21 synchronized event and RGB-D sequences, totalling 13,851 frames (7 minutes and 43 seconds of event data). Notably, 12 of these sequences feature the same object arrangement as the YCB-V subset used in the BOP challenge. Ground truth poses are generated by detecting objects in the RGB-D frames, interpolating the poses to align with the event timestamps, and then transferring them to the event coordinate frame using extrinsic calibration.

Estimating 6D object poses is a major challenge in 3D computer vision. Building on successful instance-level approaches, research is shifting towards category-level pose estimation for practical applications. Current categorylevel datasets, however, fall short in annotation quality and pose variety. Addressing this, we introduce HouseCat6D, a new category-level 6D pose dataset. It features 1) multimodality with Polarimetric RGB and Depth (RGBD+P), 2) encompasses 194 diverse objects across 10 household categories, including two photometrically challenging ones, and 3) provides high-quality pose annotations with an error range of only 1.35 mm to 1.74 mm. The dataset also includes 4) 41 large-scale scenes with comprehensive viewpoint and occlusion coverage, 5) a checkerboard-free environment, and 6) dense 6D parallel-jaw robotic grasp annotations. Additionally, we present benchmark results for leading category-level pose estimation networks.

The dataset consists of images of Human palms captured using a mobile phone. The images have been taken in a real-world scenario like holding objects or performing simple gestures. The dataset has a wide variety of variations like illumination, distances etc. It consists of images of 3 main gestures: Frontal-open palm, Back open palm and fist with the wrist. It also has a lot of images with people wearing gloves. Dataset Features: - Captured by 4000+ unique users - Covers wide variety of palm images in indoor and outdoor scenes - Images of palm with gloves - Captured by male and female - Distributed age groups like teenage, adults and old - Captured using mobile phones - Highly diverse - Various lighting conditions like day, night, indoor and outdoor - Outdoor scene with a variety of viewpoints Annotations Format: - Classification and detection annotations available - Multiple category annotations possible - COCO, PASCAL VOC and YOLO formats To download full datasets or to submit a request for your dataset needs, please ping us at (sales@datacluster.ai)[sales@datacluster.ai] Visit www.datacluster.ai to know more.