Depth Completion

KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) is one of the most popular datasets for use in mobile robotics and autonomous driving. It consists of hours of traffic scenarios recorded with a variety of sensor modalities, including high-resolution RGB, grayscale stereo cameras, and a 3D laser scanner. Despite its popularity, the dataset itself does not contain ground truth for semantic segmentation. However, various researchers have manually annotated parts of the dataset to fit their necessities. Álvarez et al. generated ground truth for 323 images from the road detection challenge with three classes: road, vertical, and sky. Zhang et al. annotated 252 (140 for training and 112 for testing) acquisitions – RGB and Velodyne scans – from the tracking challenge for ten object categories: building, sky, road, vegetation, sidewalk, car, pedestrian, cyclist, sign/pole, and fence. Ros et al. labeled 170 training images and 46 testing images (from the visual odometry challenge) with 11 classes: building, tree, sky, car, sign, road, pedestrian, fence, pole, sidewalk, and bicyclist. Source: A Review on Deep Learning Techniques Applied to Semantic Segmentation Image Source: http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d

The NYU-Depth V2 data set is comprised of video sequences from a variety of indoor scenes as recorded by both the RGB and Depth cameras from the Microsoft Kinect. It features: * 1449 densely labeled pairs of aligned RGB and depth images * 464 new scenes taken from 3 cities * 407,024 new unlabeled frames * Each object is labeled with a class and an instance number. The dataset has several components: * Labeled: A subset of the video data accompanied by dense multi-class labels. This data has also been preprocessed to fill in missing depth labels. * Raw: The raw RGB, depth and accelerometer data as provided by the Kinect. * Toolbox: Useful functions for manipulating the data and labels. Source: https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html Image Source: https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html

The Matterport3D dataset is a large RGB-D dataset for scene understanding in indoor environments. It contains 10,800 panoramic views inside 90 real building-scale scenes, constructed from 194,400 RGB-D images. Each scene is a residential building consisting of multiple rooms and floor levels, and is annotated with surface construction, camera poses, and semantic segmentation. Source: Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention

The dataset was collected using the Intel RealSense D435i camera, which was configured to produce synchronized accelerometer and gyroscope measurements at 400 Hz, along with synchronized VGA-size (640 x 480) RGB and depth streams at 30 Hz. The depth frames are acquired using active stereo and is aligned to the RGB frame using the sensor factory calibration. All the measurements are timestamped. The dataset contains 56 sequences in total, both indoor and outdoor with challenging motion. Typical scenes include classrooms, offices, stairwells, laboratories, and gardens. Of the 56 sequences, 48 sequences (approximately 47K frames) are designated for training and 8 sequences for testing, from which we sampled 800 frames to construct the testing set. Each sequence constains sparse depth maps at three density levels, 1500, 500 and 150 points, corresponding to 0.5%, 0.15% and 0.05% of VGA size.

The KITTI-Depth dataset includes depth maps from projected LiDAR point clouds that were matched against the depth estimation from the stereo cameras. The depth images are highly sparse with only 5% of the pixels available and the rest is missing. The dataset has 86k training images, 7k validation images, and 1k test set images on the benchmark server with no access to the ground truth. Source: Confidence Propagation through CNNs for Guided Sparse Depth Regression Image Source: http://www.cvlibs.net/datasets/kitti/eval_depth.php?benchmark=depth_prediction

TransCG is the first large-scale real-world dataset for transparent object depth completion and grasping, which contains 57,715 RGB-D images of 51 transparent objects and many opaque objects captured from different perspectives (~240 viewpoints) of 130 scenes under real-world settings. The samples are captured by two different types of cameras (Realsense D435 & L515). The following data is provided: - The 3D model of the transparent object; - The 6dpose of the transparent object in each viewpoint of each scene; - The raw RGB-D image, and the ground-truth refined depth image; - The mask of the transparent objects; - The ground-truth surface normals of every sample.

SuperCaustics is a simulation tool made in Unreal Engine for generating massive computer vision datasets that include transparent objects. SuperCaustics is a real-time, open-source simulation of transparent objects designed for deep learning applications. SuperCaustics features extensive modules for stochastic environment creation; uses hardware ray-tracing to support caustics, dispersion, and refraction; and enables generating massive datasets with multi-modal, pixel-perfect ground truth annotations. <p align="center"> <img src="https://github.com/MMehdiMousavi/SuperCaustics/raw/main/Assets/SuperCaustics.gif" alt="drawing" width="600"/> </p>

Bosch Industrial Depth Completion Dataset (BIDCD) is an RGBD dataset for of static table-top scenes with industrial objects. The data was collected with a RealSense depth-camera mounted on a robotic arm, i.e. from multiple Points-of-View (POV), approximately 60 for each scene. We generated depth ground truth with a customized pipeline for removing erroneous depth values, and applied Multi-View geometry to fuse the cleaned depth frames and fill-in missing information. The fused scene mesh was back-projected to each POV, and finally a bi-lateral filter was applied to reduce the remaining holes. For each scene we provide RGB, raw Depth, Ground-Truth Depth. We also provide a corresponding file-system with 3D information: our fused meshes, camera poses, and camera parameters. A simpler dataset with a Single-Item (SI) in each scene is also provided, using fewer POV, approximately 4 for each scene.

ConSLAM is a real-world dataset collected periodically on a construction site to measure the accuracy of mobile scanners' SLAM algorithms. The dataset contains time-synchronized and spatially registered RGB and NIR images and 360-deg LiDAR scans, 9-axis IMU measurements, and professional ground-truth terrestrial laser scans. This dataset reflects the periodic need to scan construction sites with the aim of accurately monitoring progress using a hand-held scanner. The sensors used for data acquisition are: - LiDAR: Velodyne VLP-16. - RGB camera: Alvium U-319c, 3.2 MP. - NIR camera: Alvium 1800 U-501, 5.0 MP. - 9-axis IMU: Xsens MTi-610.

PLAD is a dataset where sparse depth is provided by line-based visual SLAM to verify StructMDC.

Abstract: We introduce the multi-spectral stereo (MS2) outdoor dataset, including stereo RGB, stereo NIR, stereo thermal, stereo LiDAR data, and GPS/IMU information. Our dataset provides rectified and synchronized 184K data pairs taken from city, residential, road, campus, and suburban areas in the morning, daytime, and nighttime under clear-sky, cloudy, and rainy conditions. We designed the dataset to explore various computer vision algorithms from multi-spectral sensor data to achieve high-level performance, reliability, and robustness against challenging environments. MS2 dataset provides: * 1. (Synchronized) Stereo RGB images / Stereo NIR images / Stereo thermal images * 2. (Synchronized) Stereo LiDAR scans / GPS/IMU navigation data * 3. Projected depth map (in RGB, NIR, thermal image planes) * 4. Odometry data (in RGB, NIR, thermal cameras, and LiDAR coordinates)