Scene Understanding

DeepLoc is a large-scale urban outdoor localization dataset. The dataset is currently comprised of one scene spanning an area of 110 x 130 m, that a robot traverses multiple times with different driving patterns. The dataset creators use a LiDAR-based SLAM system with sub-centimeter and sub-degree accuracy to compute the pose labels that provided as groundtruth. Poses in the dataset are approximately spaced by 0.5 m which is twice as dense as other relocalization datasets. Furthermore, for each image the dataset creators provide pixel-wise semantic segmentation annotations for ten categories: Background, Sky, Road, Sidewalk, Grass, Vegetation, Building, Poles & Fences, Dynamic and Void. The dataset is divided into a train and test splits such that the train set comprises seven loops with alternating driving styles amounting to 2737 images, while the test set comprises three loops with a total of 1173 images. The dataset also contains global GPS/INS data and LiDAR measurements. This dataset can be very challenging for vision based applications such as global localization, camera relocalization, semantic segmentation, visual odometry and loop closure detection, as it contains substantial lighting, weather changes, repeating structures, reflective and transparent glass buildings. Source: http://deeploc.cs.uni-freiburg.de/ Image Source: http://deeploc.cs.uni-freiburg.de/

SynPick is a synthetic dataset for dynamic scene understanding in bin-picking scenarios. In contrast to existing datasets, this dataset is both situated in a realistic industrial application domain -- inspired by the well-known Amazon Robotics Challenge (ARC) -- and features dynamic scenes with authentic picking actions as chosen by our picking heuristic developed for the ARC 2017. The dataset is compatible with the popular BOP dataset format. The dataset consists of 21 Synthetic videos with 503,232 with diverse lightning and 3 different views of each video.

Open Images is a computer vision dataset covering ~9 million images with labels spanning thousands of object categories. A subset of 1.9M includes diverse annotations types. - 15,851,536 boxes on 600 classes - 2,785,498 instance segmentations on 350 classes - 3,284,280 relationship annotations on 1,466 relationships - 675,155 localized narratives (synchronized voice, mouse trace, and text caption) - 66,391,027 point-level annotations on 5,827 classes - 61,404,966 image-level labels on 20,638 classes Images are under a CC BY 2.0 license, annotations under CC BY 4.0 license.

A Large Dataset of Object Scans is a dataset of more than ten thousand 3D scans of real objects. To create the dataset, the authors recruited 70 operators, equipped them with consumer-grade mobile 3D scanning setups, and paid them to scan objects in their environments. The operators scanned objects of their choosing, outside the laboratory and without direct supervision by computer vision professionals. The result is a large and diverse collection of object scans: from shoes, mugs, and toys to grand pianos, construction vehicles, and large outdoor sculptures. The authors worked with an attorney to ensure that data acquisition did not violate privacy constraints. The acquired data was placed in the public domain and is available freely.

A new RGB-D video dataset, i.e., UCLA Human-Human-Object Interaction (HHOI) dataset, which includes 3 types of human-human interactions, i.e., shake hands, high-five, pull up, and 2 types of human-object-human interactions, i.e., throw and catch, and hand over a cup. On average, there are 23.6 instances per interaction performed by totally 8 actors recorded from various views. Each interaction lasts 2-7 seconds presented at 10-15 fps. Source: Learning Social Affordance for Human-Robot Interaction

LaRS is the largest and most diverse panoptic maritime obstacle detection dataset. Highlights: * Diverse scenes from manual capture, public online videos and existing datasets * USV-centric point of view * 4000+ manually per-pixel labelled frames: * 3 stuff categories and 8 thing (dynamic obstacles) categories * 20 scene-level attributes (e.g. illumination, reflections, conditions) * Temporal context for each annotated frame (9 preceding frames, total: 40k frames)

A dataset which contains over 200 apartments. Source: Rent3D: Floor-Plan Priors for Monocular Layout Estimation

The semantic line (SEL) dataset contains 1,750 outdoor images in total, which are split into 1,575 training and 175 testing images. Each semantic line is annotated by the coordinates of the two end-points on an image boundary. If an image has a single dominant line, it is set as the ground truth primary semantic line. If an image has multiple semantic lines, the line with the best rank by human annotators is set as the ground-truth primary line, and the others as additional ground-truth semantic lines. In SEL, 61% of the images contain multiple semantic lines.

VidOR (Video Object Relation) dataset contains 10,000 videos (98.6 hours) from YFCC100M collection together with a large amount of fine-grained annotations for relation understanding. In particular, 80 categories of objects are annotated with bounding-box trajectory to indicate their spatio-temporal location in the videos; and 50 categories of relation predicates are annotated among all pairs of annotated objects with starting and ending frame index. This results in around 50,000 object and 380,000 relation instances annotated. To use the dataset for model development, the dataset is split into 7,000 videos for training, 835 videos for validation, and 2,165 videos for testing.

WWW Crowd provides 10,000 videos with over 8 million frames from 8,257 diverse scenes, therefore offering a comprehensive dataset for the area of crowd understanding.

AeroRIT is a hyperspectral dataset to facilitate aerial hyperspectral scene understanding. Source: AeroRIT: A New Scene for Hyperspectral Image Analysis Image Source: https://github.com/aneesh3108/AeroRIT

CDS2K is a benchmark for Concealed scene understanding (CSU), which is a hot computer vision topic aiming to perceive objects with camouflaged properties. It is a concealed defect segmentation dataset from the five well-known defect segmentation databases. It contains five sub-databases: MVTecAD, NEU, CrackForest, KolektorSDD, and MagneticTile. The defective regions are highlighted with red rectangles. Source: Advances in Deep Concealed Scene Understanding Image Source:Advances in Deep Concealed Scene Understanding | Github Repo

The VideoNavQA dataset contains pairs of questions and videos generated in the House3D environment. The goal of this dataset is to assess question-answering performance from nearly-ideal navigation paths, while considering a much more complete variety of questions than current instantiations of the Embodied Question Answering (EQA) task. VideoNavQA contains approximately 101,000 pairs of videos and questions, 28 types of questions belonging to 8 categories, with 70 possible answers. Each question type is associated with a template that facilitates programmatic generation using ground truth information extracted from the video. The complexity of the questions in the dataset is far beyond that of other similar tasks using this generation method (such as CLEVR): the questions involve single or multiple object/room existence, object/room counting, object color recognition and localization, spatial reasoning, object/room size comparison and equality of object attributes (color, room location). Source: VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering

A large collection of interaction-rich video data which are annotated and analyzed. Source: From Lifestyle Vlogs to Everyday Interactions

A new resource to train and evaluate multitask systems on samples in multiple modalities and three languages. Source: MLM: A Benchmark Dataset for Multitask Learning with Multiple Languages and Modalities

Includes challenging sequences and extensive data stratification in-terms of camera and object motion, velocity magnitudes, direction, and rotational speeds. Source: 0-MMS: Zero-Shot Multi-Motion Segmentation With A Monocular Event Camera

The RailEye3D dataset, a collection of train-platform scenarios for applications targeting passenger safety and automation of train dispatching, consists of 10 image sequences captured at 6 railway stations in Austria. Annotations for multi-object tracking are provided in both an unified format as well as the ground-truth format used in the MOTChallenge.

InstaOrder can be used to understand the geometrical relationships of instances in an image. The dataset consists of 2.9M annotations of geometric orderings for class-labeled instances in 101K natural scenes. The scenes were annotated by 3,659 crowd-workers regarding (1) occlusion order that identifies occluder/occludee and (2) depth order that describes ordinal relations that consider relative distance from the camera.

ODMS is a dataset for learning Object Depth via Motion and Segmentation. ODMS training data are configurable and extensible, with each training example consisting of a series of object segmentation masks, camera movement distances, and ground truth object depth. As a benchmark evaluation, the dataset provides four ODMS validation and test sets with 15,650 examples in multiple domains, including robotics and driving. Source: https://github.com/griffbr/ODMS Image Source: https://github.com/griffbr/ODMS

Stanford-ECM is an egocentric multimodal dataset which comprises about 27 hours of egocentric video augmented with heart rate and acceleration data. The lengths of the individual videos cover a diverse range from 3 minutes to about 51 minutes in length. A mobile phone was used to collect egocentric video at 720x1280 resolution and 30 fps, as well as triaxial acceleration at 30Hz. The mobile phone was equipped with a wide-angle lens, so that the horizontal field of view was enlarged from 45 degrees to about 64 degrees. A wrist-worn heart rate sensor was used to capture the heart rate every 5 seconds. The phone and heart rate monitor was time-synchronized through Bluetooth, and all data was stored in the phone’s storage. Piecewise cubic polynomial interpolation was used to fill in any gaps in heart rate data. Finally, data was aligned to the millisecond level at 30 Hz. Source: http://ai.stanford.edu/~syyeung/ecm_dataset/egocentric_multimodal.html Image Source: http://ai.stanford.edu/~syyeung/ecm_dataset/egocentric_multimodal.html

A dataset containing 5000 images with 37,993 thousand relationships. The dataset contains 100 object categories and 70 predicate categories connecting those objects together. Source: Visual Relationship Detection with Language prior and Softmax

TDW is a 3D virtual world simulation platform, utilizing state-of-the-art video game engine technology. A TDW simulation consists of two components: a) the Build, a compiled executable running on the Unity3D Engine, which is responsible for image rendering, audio synthesis and physics simulations; and b) the Controller, an external Python interface to communicate with the build.

The Apron Dataset focuses on training and evaluating classification and detection models for airport-apron logistics. In addition to bounding boxes and object categories the dataset is enriched with meta parameters to quantify the models’ robustness against environmental influences.

Contains more than 5,000 images of 10,000 liquid containers in context labelled with volume, amount of content, bounding box annotation, and corresponding similar 3D CAD models. Source: See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Content

DeepLocCross is a localization dataset that contains RGB-D stereo images captured at 1280 x 720 pixels at a rate of 20 Hz. The ground-truth pose labels are generated using a LiDAR-based SLAM system. In addition to the 6-DoF localization poses of the robot, the dataset additionally contains tracked detections of the observable dynamic objects. Each tracked object is identified using a unique track ID, spatial coordinates, velocity and orientation angle. Furthermore, as the dataset contains multiple pedestrian crossings, labels at each intersection indicating its safety for crossing are provided. This dataset consists of seven training sequences with a total of 2264 images, and three testing sequences with a total of 930 images. The dynamic nature of the surrounding environment at which the dataset was captured renders the tasks of localization and visual odometry estimation extremely challenging due to the varying weather conditions, presence of shadows and motion blur caused by the movement of the robot platform. Furthermore, the presence of multiple dynamic objects often results in partial and full occlusions to the informative regions of the image. Moreover, the presence of repeated structures render the pose estimation task more challenging. Overall this dataset covers a wide range of perception related tasks such as loop closure detection, semantic segmentation, visual odometry estimation, global localization, scene flow estimation and behavior prediction. Source: http://deeploc.cs.uni-freiburg.de/ Image Source: http://deeploc.cs.uni-freiburg.de/

The dfd_indoor dataset contains 110 images for training and 29 images for testing. The dfd_outdoor dataset contains 34 images for tests; no ground truth was given for this dataset, as the depth sensor only works on indoor scenes. Source: DFD dataset

NavigationNet is a computer vision dataset and benchmark to allow the utilization of deep reinforcement learning on scene-understanding-based indoor navigation. Source: NavigationNet: A Large-scale Interactive Indoor Navigation Dataset Image Source: https://arxiv.org/pdf/1808.08374v1.pdf

Panoramic Video Panoptic Segmentation Dataset is a large-scale dataset that offers high-quality panoptic segmentation labels for autonomous driving. The dataset has labels for 28 semantic categories and 2,860 temporal sequences that were captured by five cameras mounted on autonomous vehicles driving in three different geographical locations, leading to a total of 100k labeled camera images. Source: Waymo Open Dataset: Panoramic Video Panoptic Segmentation Image Source: https://arxiv.org/pdf/2206.07704v1.pdf

Green family of datasets for emergent communications on relations. By comparison with other relations datasets, TexRel provides rapid training and experimentation, whilst being sufficiently large to avoid overfitting in the context of emergent communications.