Depth Estimation

Cityscapes is a large-scale database which focuses on semantic understanding of urban street scenes. It provides semantic, instance-wise, and dense pixel annotations for 30 classes grouped into 8 categories (flat surfaces, humans, vehicles, constructions, objects, nature, sky, and void). The dataset consists of around 5000 fine annotated images and 20000 coarse annotated ones. Data was captured in 50 cities during several months, daytimes, and good weather conditions. It was originally recorded as video so the frames were manually selected to have the following features: large number of dynamic objects, varying scene layout, and varying background. Source: A Review on Deep Learning Techniques Applied to Semantic Segmentation Image Source: https://www.cityscapes-dataset.com/dataset-overview/

KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) is one of the most popular datasets for use in mobile robotics and autonomous driving. It consists of hours of traffic scenarios recorded with a variety of sensor modalities, including high-resolution RGB, grayscale stereo cameras, and a 3D laser scanner. Despite its popularity, the dataset itself does not contain ground truth for semantic segmentation. However, various researchers have manually annotated parts of the dataset to fit their necessities. Álvarez et al. generated ground truth for 323 images from the road detection challenge with three classes: road, vertical, and sky. Zhang et al. annotated 252 (140 for training and 112 for testing) acquisitions – RGB and Velodyne scans – from the tracking challenge for ten object categories: building, sky, road, vegetation, sidewalk, car, pedestrian, cyclist, sign/pole, and fence. Ros et al. labeled 170 training images and 46 testing images (from the visual odometry challenge) with 11 classes: building, tree, sky, car, sign, road, pedestrian, fence, pole, sidewalk, and bicyclist. Source: A Review on Deep Learning Techniques Applied to Semantic Segmentation Image Source: http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d

ScanNet is an instance-level indoor RGB-D dataset that includes both 2D and 3D data. It is a collection of labeled voxels rather than points or objects. Up to now, ScanNet v2, the newest version of ScanNet, has collected 1513 annotated scans with an approximate 90% surface coverage. In the semantic segmentation task, this dataset is marked in 20 classes of annotated 3D voxelized objects. Source: A Review of Point Cloud Semantic Segmentation Image Source: http://www.scan-net.org/

The NYU-Depth V2 data set is comprised of video sequences from a variety of indoor scenes as recorded by both the RGB and Depth cameras from the Microsoft Kinect. It features: * 1449 densely labeled pairs of aligned RGB and depth images * 464 new scenes taken from 3 cities * 407,024 new unlabeled frames * Each object is labeled with a class and an instance number. The dataset has several components: * Labeled: A subset of the video data accompanied by dense multi-class labels. This data has also been preprocessed to fill in missing depth labels. * Raw: The raw RGB, depth and accelerometer data as provided by the Kinect. * Toolbox: Useful functions for manipulating the data and labels. Source: https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html Image Source: https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html

The SUN RGBD dataset contains 10335 real RGB-D images of room scenes. Each RGB image has a corresponding depth and segmentation map. As many as 700 object categories are labeled. The training and testing sets contain 5285 and 5050 images, respectively. Source: Mix and match networks: multi-domain alignment for unpaired image-to-image translation Image Source: https://rgbd.cs.princeton.edu/

The Matterport3D dataset is a large RGB-D dataset for scene understanding in indoor environments. It contains 10,800 panoramic views inside 90 real building-scale scenes, constructed from 194,400 RGB-D images. Each scene is a residential building consisting of multiple rooms and floor levels, and is annotated with surface construction, camera poses, and semantic segmentation. Source: Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention

TUM RGB-D is an RGB-D dataset. It contains the color and depth images of a Microsoft Kinect sensor along the ground-truth trajectory of the sensor. The data was recorded at full frame rate (30 Hz) and sensor resolution (640x480). The ground-truth trajectory was obtained from a high-accuracy motion-capture system with eight high-speed tracking cameras (100 Hz). Source: https://vision.in.tum.de/data/datasets/rgbd-dataset Image Source: https://vision.in.tum.de/research/rgb-d_sensors_kinect

The Middlebury Stereo dataset consists of high-resolution stereo sequences with complex geometry and pixel-accurate ground-truth disparity data. The ground-truth disparities are acquired using a novel technique that employs structured lighting and does not require the calibration of the light projectors. Source: https://vision.middlebury.edu/stereo/data/ Image Source: https://www.researchgate.net/figure/The-stereo-matching-results-on-the-Middlebury-dataset-From-left-to-right-each-set-of_fig3_273399625

SUNCG is a large-scale dataset of synthetic 3D scenes with dense volumetric annotations. The dataset is currently not available. Source: https://sscnet.cs.princeton.edu/ Image Source: https://sscnet.cs.princeton.edu/

The MegaDepth dataset is a dataset for single-view depth prediction that includes 196 different locations reconstructed from COLMAP SfM/MVS. Source: MegaDepth: Learning Single-View Depth Prediction from Internet Photos

The 2D-3D-S dataset provides a variety of mutually registered modalities from 2D, 2.5D and 3D domains, with instance-level semantic and geometric annotations. It covers over 6,000 m2 collected in 6 large-scale indoor areas that originate from 3 different buildings. It contains over 70,000 RGB images, along with the corresponding depths, surface normals, semantic annotations, global XYZ images (all in forms of both regular and 360° equirectangular images) as well as camera information. It also includes registered raw and semantically annotated 3D meshes and point clouds. The dataset enables development of joint and cross-modal learning models and potentially unsupervised approaches utilizing the regularities present in large-scale indoor spaces. Source: https://github.com/alexsax/2D-3D-Semantics Image Source: https://github.com/alexsax/2D-3D-Semantics

Taskonomy provides a large and high-quality dataset of varied indoor scenes. - Complete pixel-level geometric information via aligned meshes. - Semantic information via knowledge distillation from ImageNet, MS COCO, and MIT Places. - Globally consistent camera poses. Complete camera intrinsics. - High-definition images. - 3x times big as ImageNet. Source: Taskonomy Image Source: http://taskonomy.stanford.edu/

The Replay-Attack Database for face spoofing consists of 1300 video clips of photo and video attack attempts to 50 clients, under different lighting conditions. All videos are generated by either having a (real) client trying to access a laptop through a built-in webcam or by displaying a photo or a video recording of the same client for at least 9 seconds. Source: https://www.idiap.ch/dataset/replayattack Image Source: https://www.researchgate.net/figure/Sample-images-from-the-PRINT-ATTACK-1-and-REPLAY-ATTACK-12-databases-Top-and-bottom_fig1_330400888

Virtual KITTI is a photo-realistic synthetic video dataset designed to learn and evaluate computer vision models for several video understanding tasks: object detection and multi-object tracking, scene-level and instance-level semantic segmentation, optical flow, and depth estimation. Virtual KITTI contains 50 high-resolution monocular videos (21,260 frames) generated from five different virtual worlds in urban settings under different imaging and weather conditions. These worlds were created using the Unity game engine and a novel real-to-virtual cloning method. These photo-realistic synthetic videos are automatically, exactly, and fully annotated for 2D and 3D multi-object tracking and at the pixel level with category, instance, flow, and depth labels (cf. below for download links). Source: https://europe.naverlabs.com/research/computer-vision/proxy-virtual-worlds-vkitti-1/ Image Source: https://arxiv.org/pdf/1605.06457.pdf

The Make3D dataset is a monocular Depth Estimation dataset that contains 400 single training RGB and depth map pairs, and 134 test samples. The RGB images have high resolution, while the depth maps are provided at low resolution. Source: Structured Coupled Generative Adversarial Networks for Unsupervised Monocular Depth Estimation Image Source: http://make3d.cs.cornell.edu/data.html#make3d

SUN3D contains a large-scale RGB-D video database, with 8 annotated sequences. Each frame has a semantic segmentation of the objects in the scene and information about the camera pose. It is composed by 415 sequences captured in 254 different spaces, in 41 different buildings. Moreover, some places have been captured multiple times at different moments of the day. Source: A Review on Deep Learning TechniquesApplied to Semantic Segmentation Image Source: http://sun3d.cs.princeton.edu/

ETHD is a multi-view stereo benchmark / 3D reconstruction benchmark that covers a variety of indoor and outdoor scenes. Ground truth geometry has been obtained using a high-precision laser scanner. A DSLR camera as well as a synchronized multi-camera rig with varying field-of-view was used to capture images. Source: A Multi-View Stereo Benchmark With High-Resolution Images and Multi-Camera Videos

For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. Hypersim is a photorealistic synthetic dataset for holistic indoor scene understanding. It contains 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry. Source: https://github.com/apple/ml-hypersim Image Source: https://github.com/apple/ml-hypersim

The MSU-MFSD dataset contains 280 video recordings of genuine and attack faces. 35 individuals have participated in the development of this database with a total of 280 videos. Two kinds of cameras with different resolutions (720×480 and 640×480) were used to record the videos from the 35 individuals. For the real accesses, each individual has two video recordings captured with the Laptop cameras and Android, respectively. For the video attacks, two types of cameras, the iPhone and Canon cameras were used to capture high definition videos on each of the subject. The videos taken with Canon camera were then replayed on iPad Air screen to generate the HD replay attacks while the videos recorded by the iPhone mobile were replayed itself to generate the mobile replay attacks. Photo attacks were produced by printing the 35 subjects’ photos on A3 papers using HP colour printer. The recording videos with respect to the 35 individuals were divided into training (15 subjects with 120 videos) and testing (40 subjects with 160 videos) datasets, respectively. Source: Enhance the Motion Cues for Face Anti-Spoofing using CNN-LSTM Architecture Image Source: face anti-spoofing based on color texture analysis

Diode Dense Indoor/Outdoor DEpth (DIODE) is the first standard dataset for monocular depth estimation comprising diverse indoor and outdoor scenes acquired with the same hardware setup. The training set consists of 8574 indoor and 16884 outdoor samples from 20 scans each. The validation set contains 325 indoor and 446 outdoor samples with each set from 10 different scans. The ground truth density for the indoor training and validation splits are approximately 99.54% and 99%, respectively. The density of the outdoor sets are naturally lower with 67.19% for training and 78.33% for validation subsets. The indoor and outdoor ranges for the dataset are 50m and 300m, respectively. Source: Bidirectional Attention Network for Monocular Depth Estimation Image Source: https://diode-dataset.org/

DDAD is a new autonomous driving benchmark from TRI (Toyota Research Institute) for long range (up to 250m) and dense depth estimation in challenging and diverse urban conditions. It contains monocular videos and accurate ground-truth depth (across a full 360 degree field of view) generated from high-density LiDARs mounted on a fleet of self-driving cars operating in a cross-continental setting. DDAD contains scenes from urban settings in the United States (San Francisco, Bay Area, Cambridge, Detroit, Ann Arbor) and Japan (Tokyo, Odaiba). Source: https://github.com/TRI-ML/DDAD Image Source: https://github.com/TRI-ML/DDAD

The Middlebury 2014 dataset contains a set of 23 high resolution stereo pairs for which known camera calibration parameters and ground truth disparity maps obtained with a structured light scanner are available. The images in the Middlebury dataset all show static indoor scenes with varying difficulties including repetitive structures, occlusions, wiry objects as well as untextured areas. Source: Using Self-Contradiction to Learn Confidence Measures in Stereo Vision Image Source: https://vision.middlebury.edu/stereo/data/scenes2014/

CASIA-MFSD is a dataset for face anti-spoofing. It contains 50 subjects, and 12 videos for each subject under different resolutions and light conditions. Three different spoof attacks are designed: replay, warp print and cut print attacks. The database contains 600 video recordings, in which 240 videos of 20 subjects are used for training and 360 videos of 30 subjects for testing. Source: Improving Face Anti-Spoofing by 3D Virtual Synthesis Image Source: https://link.springer.com/referenceworkentry/10.1007%2F978-1-4899-7488-4_9067

Virtual KITTI 2 is an updated version of the well-known Virtual KITTI dataset which consists of 5 sequence clones from the KITTI tracking benchmark. In addition, the dataset provides different variants of these sequences such as modified weather conditions (e.g. fog, rain) or modified camera configurations (e.g. rotated by 15◦). For each sequence we provide multiple sets of images containing RGB, depth, class segmentation, instance segmentation, flow, and scene flow data. Camera parameters and poses as well as vehicle locations are available as well. In order to showcase some of the dataset’s capabilities, we ran multiple relevant experiments using state-of-the-art algorithms from the field of autonomous driving. The dataset is available for download at https://europe.naverlabs.com/Research/Computer-Vision/Proxy-Virtual-Worlds.

SiW provides live and spoof videos from 165 subjects. For each subject, we have 8 live and up to 20 spoof videos, in total 4,478 videos. All videos are in 30 fps, about 15 second length, and 1080P HD resolution. The live videos are collected in four sessions with variations of distance, pose, illumination and expression. The spoof videos are collected with several attacks such as printed paper and replay.

DrivingStereo contains over 180k images covering a diverse set of driving scenarios, which is hundreds of times larger than the KITTI Stereo dataset. High-quality labels of disparity are produced by a model-guided filtering strategy from multi-frame LiDAR points. Source: DrivingStereo: A Large-Scale Dataset for Stereo Matching in Autonomous Driving Scenarios Image Source: https://drivingstereo-dataset.github.io/

DENSE (Depth Estimation oN Synthetic Events) is a new dataset with synthetic events and perfect ground truth. Source: Learning Monocular Dense Depth from Events

CASIA-FASD is a small face anti-spoofing dataset containing 50 subjects. Source: Learning Generalizable and Identity-Discriminative Representations for Face Anti-Spoofing Image Source: https://arxiv.org/abs/1511.06316

iBims-1 (independent Benchmark images and matched scans - version 1) is a new high-quality RGB-D dataset, especially designed for testing single-image depth estimation (SIDE) methods. A customized acquisition setup, composed of a digital single-lens reflex (DSLR) camera and a high-precision laser scanner was used to acquire high-resolution images and highly accurate depth maps of diverse indoors scenarios. Compared to related RGB-D datasets, iBims-1 stands out due to a very low noise level, sharp depth transitions, no occlusions, and high depth ranges. Our dataset consists of the following components: Core dataset: - 100 RGB-D image pairs of various indoor scenes in high- and low resolution - Masks for invalid, transparent and planar regions (tables, floors, walls) - Masks for distinct depth transitions - Camera calibration parameters Auxiliary dataset: - 56 different color and geometric augmentations for each image of the core dataset - Additional hand-held images for testing MVS methods - Images of printed patterns and photos posted on a wall to assess performance of textured planar surfaces - Several RGB-D image sequences of static scenes with varying illumation Source: Evaluation of CNN-based Single-Image Depth Estimation Methods Image source: https://arxiv.org/pdf/1805.01328v1.pdf

Spring is a large, high-resolution and high-detail, computer-generated benchmark for scene flow, optical flow, and stereo. Based on rendered scenes from the open-source Blender movie "Spring", it provides photo-realistic HD datasets with state-of-the-art visual effects and ground truth training data. Source: Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo Image Source: https://arxiv.org/pdf/2303.01943v1.pdf

CelebA-Spoof is a large-scale face anti-spoofing dataset with the following properties: 1. Quantity: CelebA-Spoof comprises of 625,537 pictures of 10,177 subjects, significantly larger than the existing datasets. 2. Diversity: The spoof images are captured from 8 scenes (2 environments * 4 illumination conditions) with more than 10 sensors. 3. Annotation Richness: CelebA-Spoof contains 10 spoof type annotations, as well as the 40 attribute annotations inherited from the original CelebA dataset. Source: CelebA-Spoof: Large-Scale Face Anti-Spoofing Dataset with Rich Annotations

The MannequinChallenge Dataset (MQC) provides in-the-wild videos of people in static poses while a hand-held camera pans around the scene. The dataset consists of three splits for training, validation and testing. Source: Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild Image Source: https://google.github.io/mannequinchallenge/www/index.html

A dataset for single-image 3D in the wild consisting of annotations of detailed 3D geometry for 140,000 images. Source: OASIS: A Large-Scale Dataset for Single Image 3D in the Wild

2D-3D Match Dataset is a new dataset of 2D-3D correspondences by leveraging the availability of several 3D datasets from RGB-D scans. Specifically, the data from SceneNN and 3DMatch are used. The training dataset consists of 110 RGB-D scans, of which 56 scenes are from SceneNN and 54 scenes are from 3DMatch. The 2D-3D correspondence data is generated as follows. Given a 3D point which is randomly sampled from a 3D point cloud, a set of 3D patches from different scanning views are extracted. To find a 2D-3D correspondence, for each 3D patch, its 3D position is re-projected into all RGB-D frames for which the point lies in the camera frustum, taking occlusion into account. The corresponding local 2D patches around the re-projected point are extracted. In total, around 1.4 millions 2D-3D correspondences are collected. Source: 2D-3D Match Dataset

The dataset was collected using the Intel RealSense D435i camera, which was configured to produce synchronized accelerometer and gyroscope measurements at 400 Hz, along with synchronized VGA-size (640 x 480) RGB and depth streams at 30 Hz. The depth frames are acquired using active stereo and is aligned to the RGB frame using the sensor factory calibration. All the measurements are timestamped. The dataset contains 56 sequences in total, both indoor and outdoor with challenging motion. Typical scenes include classrooms, offices, stairwells, laboratories, and gardens. Of the 56 sequences, 48 sequences (approximately 47K frames) are designated for training and 8 sequences for testing, from which we sampled 800 frames to construct the testing set. Each sequence constains sparse depth maps at three density levels, 1500, 500 and 150 points, corresponding to 0.5%, 0.15% and 0.05% of VGA size.

Dataset for face anti-spoofing in terms of both subjects and modalities. Specifically, it consists of subjects with videos and each sample has modalities (i.e., RGB, Depth and IR). Source: CASIA-SURF: A Large-scale Multi-modal Benchmark for Face Anti-spoofing