Scene Segmentation

ScanNet is an instance-level indoor RGB-D dataset that includes both 2D and 3D data. It is a collection of labeled voxels rather than points or objects. Up to now, ScanNet v2, the newest version of ScanNet, has collected 1513 annotated scans with an approximate 90% surface coverage. In the semantic segmentation task, this dataset is marked in 20 classes of annotated 3D voxelized objects. Source: A Review of Point Cloud Semantic Segmentation Image Source: http://www.scan-net.org/

The NYU-Depth V2 data set is comprised of video sequences from a variety of indoor scenes as recorded by both the RGB and Depth cameras from the Microsoft Kinect. It features: * 1449 densely labeled pairs of aligned RGB and depth images * 464 new scenes taken from 3 cities * 407,024 new unlabeled frames * Each object is labeled with a class and an instance number. The dataset has several components: * Labeled: A subset of the video data accompanied by dense multi-class labels. This data has also been preprocessed to fill in missing depth labels. * Raw: The raw RGB, depth and accelerometer data as provided by the Kinect. * Toolbox: Useful functions for manipulating the data and labels. Source: https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html Image Source: https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html

The SUN RGBD dataset contains 10335 real RGB-D images of room scenes. Each RGB image has a corresponding depth and segmentation map. As many as 700 object categories are labeled. The training and testing sets contain 5285 and 5050 images, respectively. Source: Mix and match networks: multi-domain alignment for unpaired image-to-image translation Image Source: https://rgbd.cs.princeton.edu/

The first RGB-Thermal urban scene image dataset with pixel-level annotation. We published this new RGB-Thermal semantic segmentation dataset in support of further development of autonomous vehicles in the future. This dataset contains 1569 images (820 taken at daytime and 749 taken at nighttime). Eight classes of obstacles commonly encountered during driving (car, person, bike, curve, car stop, guardrail, color cone, and bump) are labeled in this dataset.

MovieNet is a holistic dataset for movie understanding. MovieNet contains 1,100 movies with a large amount of multi-modal data, e.g. trailers, photos, plot descriptions, etc.. Besides, different aspects of manual annotations are provided in MovieNet, including 1.1M characters with bounding boxes and identities, 42K scene boundaries, 2.5K aligned description sentences, 65K tags of place and action, and 92 K tags of cinematic style.

UAVid is a high-resolution UAV semantic segmentation dataset as a complement, which brings new challenges, including large scale variation, moving object recognition and temporal consistency preservation. The UAV dataset consists of 30 video sequences capturing 4K high-resolution images in slanted views. In total, 300 images have been densely labeled with 8 classes for the semantic labeling task. Source: UAVid: A Semantic Segmentation Dataset for UAV Imagery Image Source: https://uavid.nl/

PST900 is a dataset of 894 synchronized and calibrated RGB and Thermal image pairs with per pixel human annotations across four distinct classes from the DARPA Subterranean Challenge. Source: https://arxiv.org/abs/1909.10980 Image Source: https://github.com/ShreyasSkandanS/pst900_thermal_rgb

StreetHazards is a synthetic dataset for anomaly detection, created by inserting a diverse array of foreign objects into driving scenes and re-render the scenes with these novel objects. Source: Scaling Out-of-Distribution Detection for Real-World Settings

We construct the first large-scale dataset, USIS10K, for the underwater salient instance segmentation task, which contains 10,632 images and pixel-level annotations of 7 categories. As far as we know, this is the largest salient instance segmentation dataset, and includes Class-Agnostic and Multi-Class labels simultaneously. If you use this dataset in your work, please consider to cite: @inproceedings{ lian2024icml, title={Diving into Underwater: Segment Anything Model Guided Underwater Salient Instance Segmentation and A Large-scale Dataset}, author={Shijie Lian and Ziyi Zhang and Hua Li and Wenjie Li and Laurence Tianruo Yang and Sam Kwong and Runmin Cong}, booktitle={Forty-first International Conference on Machine Learning}, year={2024}, url={https://openreview.net/forum?id=snhurpZt63} }

Mila Simulated Floods Dataset is a 1.5 square km virtual world using the Unity3D game engine including urban, suburban and rural areas. The urban environment contains skyscrapers, large buildings, and roads, as well as objects such as traffic items and vehicles. The rural environment consists of a landscape of grassy hills, forests, and mountains, with sparse houses and other buildings such as a church, and no roads. The rural and urban areas make up for 1 square km of our virtual world. The suburban environment is a residential area of 0.5 square km with many individual houses with front yards. To gather the simulated dataset, we captured before and after flood pairs from 2000 viewpoints with the following modalities: - non-flooded RGB image, depth map, segmentation map - flooded RGB image, binary mask of the flooded area, segmentation map The camera was placed about 1.5 above ground, and has a field of view of 120 degree, and the resolution of the images is 1200 x 900. At each viewpoint, we took 10 pictures, by varying slightly the position of the camera in order to augment the dataset.

A dataset comprised of real driving videos and GPS/IMU data. The BDDV dataset contains diverse driving scenarios including cities, highways, towns, and rural areas in several major cities in US. Source: End-to-end Learning of Driving Models from Large-scale Video Datasets

Charlotte-ThermalFace is a thermal face dataset. The data is fully annotated with the facial landmarks, ambient temperature, relative humidity, the air speed of the room, distance to the camera, and subject thermal sensation at the time of capturing each image. There are approximately 10,000 infrared thermal images from 10 subjects in varying thermal conditions, at several distances from the camera, and at changing head positions. We have also controlled the air temperature to change from 20.5°C ( 69°F) to 26.5 °C( 80°F). Images are available in four different temperatures, 10 relative distances from the camera, starting at 1m ( 3.3 ft) to 6.6m( 21.6 ft), and 25 head positions. • The first public facial thermal dataset annotated with the environmental properties including air temperature, relative humidity, airspeed, distance from the camera, and subjective thermal sensation of each person at the time. • All the images are manually annotated with 72 or 43 facial landmarks. • We are publishing the data in the original 16-bit radiometric TemperatureLinear format, which has the thermal value of each pixel.

Daily Activity Recordings for Artificial Intelligence (DARai, pronounced "Dahr-ree") is a multimodal, hierarchically annotated dataset constructed to understand human activities in real-world settings. DARai consists of continuous scripted and unscripted recordings of 50 participants in 10 different environments, totaling over 200 hours of data from 20 sensors including multiple camera views, depth and radar sensors, wearable inertial measurement units (IMUs), electromyography (EMG), insole pressure sensors, biomonitor sensors, and gaze tracker. To capture the complexity in human activities, DARai is annotated at three levels of hierarchy: (i) high-level activities (L1) that are independent tasks, (ii) lower-level actions (L2) that are patterns shared between activities, and (iii) fine-grained procedures (L3) that detail the exact execution steps for actions. The dataset annotations and recordings are designed so that 22.7% of L2 actions are shared between L1 activities and 14.2% of L3 procedures are shared between L2 actions. The overlap and unscripted nature of DARai allows counterfactual activities in the dataset. Experiments with various machine learning models showcase the value of DARai in uncovering important challenges in human-centered applications. Specifically, we conduct unimodal and multimodal sensor fusion experiments for recognition, temporal localization, and future action anticipation across all hierarchical annotation levels. To highlight the limitations of individual sensors, we also conduct domain-variant experiments that are enabled by DARai's multi-sensor and counterfactual activity design setup. The code, documentation, and dataset are available at the dedicated DARai website.

Abstract: We introduce the multi-spectral stereo (MS2) outdoor dataset, including stereo RGB, stereo NIR, stereo thermal, stereo LiDAR data, and GPS/IMU information. Our dataset provides rectified and synchronized 184K data pairs taken from city, residential, road, campus, and suburban areas in the morning, daytime, and nighttime under clear-sky, cloudy, and rainy conditions. We designed the dataset to explore various computer vision algorithms from multi-spectral sensor data to achieve high-level performance, reliability, and robustness against challenging environments. MS2 dataset provides: * 1. (Synchronized) Stereo RGB images / Stereo NIR images / Stereo thermal images * 2. (Synchronized) Stereo LiDAR scans / GPS/IMU navigation data * 3. Projected depth map (in RGB, NIR, thermal image planes) * 4. Odometry data (in RGB, NIR, thermal cameras, and LiDAR coordinates)