Scene Parsing

MLe2 is a dataset for the evaluation of scene text end-to-end reading systems and all intermediate stages such as text detection, script identification and text recognition. The dataset contains a total of 711 scene images covering four different scripts (Latin, Chinese, Kannada, and Hangul). Source: A fine-grained approach to scene text script identification

Open Images is a computer vision dataset covering ~9 million images with labels spanning thousands of object categories. A subset of 1.9M includes diverse annotations types. - 15,851,536 boxes on 600 classes - 2,785,498 instance segmentations on 350 classes - 3,284,280 relationship annotations on 1,466 relationships - 675,155 localized narratives (synchronized voice, mouse trace, and text caption) - 66,391,027 point-level annotations on 5,827 classes - 61,404,966 image-level labels on 20,638 classes Images are under a CC BY 2.0 license, annotations under CC BY 4.0 license.

A Large Dataset of Object Scans is a dataset of more than ten thousand 3D scans of real objects. To create the dataset, the authors recruited 70 operators, equipped them with consumer-grade mobile 3D scanning setups, and paid them to scan objects in their environments. The operators scanned objects of their choosing, outside the laboratory and without direct supervision by computer vision professionals. The result is a large and diverse collection of object scans: from shoes, mugs, and toys to grand pianos, construction vehicles, and large outdoor sculptures. The authors worked with an attorney to ensure that data acquisition did not violate privacy constraints. The acquired data was placed in the public domain and is available freely.

A new RGB-D video dataset, i.e., UCLA Human-Human-Object Interaction (HHOI) dataset, which includes 3 types of human-human interactions, i.e., shake hands, high-five, pull up, and 2 types of human-object-human interactions, i.e., throw and catch, and hand over a cup. On average, there are 23.6 instances per interaction performed by totally 8 actors recorded from various views. Each interaction lasts 2-7 seconds presented at 10-15 fps. Source: Learning Social Affordance for Human-Robot Interaction

LaRS is the largest and most diverse panoptic maritime obstacle detection dataset. Highlights: * Diverse scenes from manual capture, public online videos and existing datasets * USV-centric point of view * 4000+ manually per-pixel labelled frames: * 3 stuff categories and 8 thing (dynamic obstacles) categories * 20 scene-level attributes (e.g. illumination, reflections, conditions) * Temporal context for each annotated frame (9 preceding frames, total: 40k frames)

PGDP5K is a dataset consisting of 5000 diagram samples composed of 16 shapes, covering 5 positional relations, 22 symbol types and 6 text types, labeled with more fine-grained annotations at primitive level, including primitive classes, locations and relationships, where 1,813 non-duplicated images are selected from the Geometry3K dataset and other 3,187 images are collected from three popular textbooks across grades 6-12 on mathematics curriculum websites by taking screenshots from PDF books.

A dataset which contains over 200 apartments. Source: Rent3D: Floor-Plan Priors for Monocular Layout Estimation

The semantic line (SEL) dataset contains 1,750 outdoor images in total, which are split into 1,575 training and 175 testing images. Each semantic line is annotated by the coordinates of the two end-points on an image boundary. If an image has a single dominant line, it is set as the ground truth primary semantic line. If an image has multiple semantic lines, the line with the best rank by human annotators is set as the ground-truth primary line, and the others as additional ground-truth semantic lines. In SEL, 61% of the images contain multiple semantic lines.

VidOR (Video Object Relation) dataset contains 10,000 videos (98.6 hours) from YFCC100M collection together with a large amount of fine-grained annotations for relation understanding. In particular, 80 categories of objects are annotated with bounding-box trajectory to indicate their spatio-temporal location in the videos; and 50 categories of relation predicates are annotated among all pairs of annotated objects with starting and ending frame index. This results in around 50,000 object and 380,000 relation instances annotated. To use the dataset for model development, the dataset is split into 7,000 videos for training, 835 videos for validation, and 2,165 videos for testing.

WWW Crowd provides 10,000 videos with over 8 million frames from 8,257 diverse scenes, therefore offering a comprehensive dataset for the area of crowd understanding.

AeroRIT is a hyperspectral dataset to facilitate aerial hyperspectral scene understanding. Source: AeroRIT: A New Scene for Hyperspectral Image Analysis Image Source: https://github.com/aneesh3108/AeroRIT

CDS2K is a benchmark for Concealed scene understanding (CSU), which is a hot computer vision topic aiming to perceive objects with camouflaged properties. It is a concealed defect segmentation dataset from the five well-known defect segmentation databases. It contains five sub-databases: MVTecAD, NEU, CrackForest, KolektorSDD, and MagneticTile. The defective regions are highlighted with red rectangles. Source: Advances in Deep Concealed Scene Understanding Image Source:Advances in Deep Concealed Scene Understanding | Github Repo

The ChineseLP dataset contains 411 vehicle images (mostly of passenger cars) with Chinese license plates (LPs). It consists of 252 images captured by the authors and 159 images downloaded from the internet. The images present great variations in resolution (from 143 × 107 to 2048 × 1536 pixels), illumination and background.

PRO-teXt is an extension of PROXD with the inclusion of text prompts to synthesize objects. There are 180/20 interactions for training/testing in PRO-teXt. Each interaction involves a linguistic command corresponding to an existing room arrangement.

The VideoNavQA dataset contains pairs of questions and videos generated in the House3D environment. The goal of this dataset is to assess question-answering performance from nearly-ideal navigation paths, while considering a much more complete variety of questions than current instantiations of the Embodied Question Answering (EQA) task. VideoNavQA contains approximately 101,000 pairs of videos and questions, 28 types of questions belonging to 8 categories, with 70 possible answers. Each question type is associated with a template that facilitates programmatic generation using ground truth information extracted from the video. The complexity of the questions in the dataset is far beyond that of other similar tasks using this generation method (such as CLEVR): the questions involve single or multiple object/room existence, object/room counting, object color recognition and localization, spatial reasoning, object/room size comparison and equality of object attributes (color, room location). Source: VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering

A large collection of interaction-rich video data which are annotated and analyzed. Source: From Lifestyle Vlogs to Everyday Interactions

We introduce a large-scale image dataset EasyPortrait for portrait segmentation and face parsing. Proposed dataset can be used in several tasks, such as background removal in conference applications, teeth whitening, face skin enhancement, red eye removal or eye colorization, and so on. EasyPortrait dataset size is about 26GB, and it contains 20 000 RGB images with high quality annotated masks. This dataset is divided into training set, validation set and test set by hashed subject user_id. The training set includes 14000 images, the validation set includes 2000 images, and the test set includes 4000 images. Training images were received from 5,947 unique users, while validation was from 860 and testing was from 1,570. On average, each EasyPortrait image has 254 polygon points, from which it can be concluded that the annotation is of high quality. Segmentation masks were created from polygons for each annotation. Annotations are presented as 2D-arrays, images in *.png format with several classes: | Index | Class | |------:|:-----------| | 0 | BACKGROUND | | 1 | PERSON | | 2 | SKIN | | 3 | LEFT BROW | | 4 | RIGHT_BROW | | 5 | LEFT_EYE | | 6 | RIGHT_EYE | | 7 | LIPS | | 8 | TEETH | Also, we provide some additional meta-information for dataset in annotations/meta.zip file: | | attachment_id | user_id | data_hash | width | height | brightness | train | test | valid | |---:|:--------------|:--------|:----------|------:|-------:|-----------:|:------|:------|:------| | 0 | de81cc1c-... | 1b... | e8f... | 1440 | 1920 | 136 | True | False | False | | 1 | 3c0cec5a-... | 64... | df5... | 1440 | 1920 | 148 | False | False | True | | 2 | d17ca986-... | cf... | a69... | 1920 | 1080 | 140 | False | True | False | where: - attachment_id - image file name without extension - user_id - unique anonymized user ID - data_hash - image hash by using Perceptual hashing - width - image width - height - image height - brightness - image brightness - train, test, valid are the binary columns for train / test / val subsets respectively

An annotated dataset is released to enable dynamic scene classification that includes 80 hours of diverse high quality driving video data clips collected in the San Francisco Bay area. The dataset includes temporal annotations for road places, road types, weather, and road surface conditions. Source: Dynamic Traffic Scene Classification with Space-Time Coherence

A new resource to train and evaluate multitask systems on samples in multiple modalities and three languages. Source: MLM: A Benchmark Dataset for Multitask Learning with Multiple Languages and Modalities

Operating rooms (ORs) are complex, high-stakes environments requiring precise understanding of interactions among medical staff, tools, and equipment for enhancing surgical assistance, situational awareness, and patient safety. Current datasets fall short in scale, realism and do not capture the multimodal nature of OR scenes, limiting progress in OR modeling. To this end, we introduce MM-OR, a realistic and large-scale multimodal spatiotemporal OR dataset, and the first dataset to enable multimodal scene graph generation. MM-OR captures comprehensive OR scenes containing RGB-D data, detail views, audio, speech transcripts, robotic logs, and tracking data and is annotated with panoptic segmentations, semantic scene graphs, and downstream task labels. Further, we propose MM2SG, the first multimodal large vision-language model for scene graph generation, and through extensive experiments, demonstrate its ability to effectively leverage multimodal inputs. Together, MM-OR and MM2SG establish a new benchmark for holistic OR understanding, and open the path towards multimodal scene analysis in complex, high-stakes environments. Paper: https://arxiv.org/abs/2503.02579

Includes challenging sequences and extensive data stratification in-terms of camera and object motion, velocity magnitudes, direction, and rotational speeds. Source: 0-MMS: Zero-Shot Multi-Motion Segmentation With A Monocular Event Camera

The RailEye3D dataset, a collection of train-platform scenarios for applications targeting passenger safety and automation of train dispatching, consists of 10 image sequences captured at 6 railway stations in Austria. Annotations for multi-object tracking are provided in both an unified format as well as the ground-truth format used in the MOTChallenge.

UNDD consists of 7125 unlabelled day and night images; additionally, it has 75 night images with pixel-level annotations having classes equivalent to Cityscapes dataset. Source: UNDD Image Source: https://github.com/sauradip/night_image_semantic_segmentation

Vehicle-Rear is a novel dataset for vehicle identification that contains more than three hours of high-resolution videos, with accurate information about the make, model, color and year of nearly 3,000 vehicles, in addition to the position and identification of their license plates.

iBugMask is an in-the-wild face parsing dataset that contains 1,000 challenging face images and manually annotated labels for 11 semantic classes: background, facial skin, left/right brow, left/right eye, nose, upper/lower lip, inner mouth, and hair. The images are curated from challenging in-the-wild face alignment datasets, including 300W and Menpo. Compared with the existing face parsing datasets, iBugMask contains in-the-wild scenarios such as “party” and “conference”, which include more challenging appearance variations or multiple faces. There is a larger number of profile faces. More expressions other than ”neutral” and ”smile” are also included (e.g. ”surprise” and ”scream”). The dataset can be downloaded on here.

InstaOrder can be used to understand the geometrical relationships of instances in an image. The dataset consists of 2.9M annotations of geometric orderings for class-labeled instances in 101K natural scenes. The scenes were annotated by 3,659 crowd-workers regarding (1) occlusion order that identifies occluder/occludee and (2) depth order that describes ordinal relations that consider relative distance from the camera.

• 5 domains: synthetic domain, document domain, street view domain, handwritten domain, and car license domain * over five million images

The Mobile Turkish Scene Text (MTST 200) dataset consists of 200 indoor and outdoor Turkish scene text images. The images were collected with mobile phones and downsized to 576 × 1024 (portrait) or 1024 × 576 (landscape) pixels. The text lines are horizontal or near horizontal, some with slight in– and out-of-plane rotations.

ODMS is a dataset for learning Object Depth via Motion and Segmentation. ODMS training data are configurable and extensible, with each training example consisting of a series of object segmentation masks, camera movement distances, and ground truth object depth. As a benchmark evaluation, the dataset provides four ODMS validation and test sets with 15,650 examples in multiple domains, including robotics and driving. Source: https://github.com/griffbr/ODMS Image Source: https://github.com/griffbr/ODMS

Rent3D++ is an extension of the Rent3D floorplans + photos dataset. The floorplans are annotated with room outline polygons, doors/windows as line segments, object-icons as axis-aligned bounding boxes, room-door-room connectivity graphs, and photo-room assignments. We have extracted rectified surface crops from architectural surfaces in photos, and these can drive interior texturing/material modeling tasks. This dataset can be used with our paper Plan2Scene to generate textured 3D mesh models of houses using floorplans and photos. The complete list of improvements we did on the Rent3D dataset are as follows: - Fixed incorrectly categorized rooms and added wall outlines and categories from missing rooms. - Expanding the room category set {reception, bedroom, kitchen, bathroom, outdoor} by adding another 7 common room types: {closet, entrance, corridor, staircase, balcony, terrace, unknown}. - Generated room-door-room connectivity graphs for floorplans. - Annotated all windows, doors, and other wall openings, and associated them with corresponding rooms. - Defined a new 60/20/20% (129/43/43 houses) training, validation, test split (cf. original 100/30/85 house split), giving more samples to training and validation. - Extract rectified surface crops from architectural surfaces seen in photos (floors, walls, ceilings). - Annotated axis-aligned bounding boxes for fixed object icons indicated on the test set floorplans.

Stanford-ECM is an egocentric multimodal dataset which comprises about 27 hours of egocentric video augmented with heart rate and acceleration data. The lengths of the individual videos cover a diverse range from 3 minutes to about 51 minutes in length. A mobile phone was used to collect egocentric video at 720x1280 resolution and 30 fps, as well as triaxial acceleration at 30Hz. The mobile phone was equipped with a wide-angle lens, so that the horizontal field of view was enlarged from 45 degrees to about 64 degrees. A wrist-worn heart rate sensor was used to capture the heart rate every 5 seconds. The phone and heart rate monitor was time-synchronized through Bluetooth, and all data was stored in the phone’s storage. Piecewise cubic polynomial interpolation was used to fill in any gaps in heart rate data. Finally, data was aligned to the millisecond level at 30 Hz. Source: http://ai.stanford.edu/~syyeung/ecm_dataset/egocentric_multimodal.html Image Source: http://ai.stanford.edu/~syyeung/ecm_dataset/egocentric_multimodal.html

This dataset contains 2,000 images taken from inside a warehouse of the Energy Company of Paraná (Copel), which directly serves more than 4 million consuming units in the Brazilian state of Paraná. The images were acquired with three different cameras and are available in the JPG format with a resolution between 2,340 × 4,160 and 3,120 × 4,160 pixels. The dataset is split into three sets: training (800 images), validation (400 images) and testing (800 images). Every image has the following annotations available in a text file: the camera in which the image was taken, the counter’s position (x,y,w,h) and reading, as well as the position of each digit. All counters of the dataset (regardless of meter type) have 5 digits, and thus 10,000 digits were manually annotated. Source: Convolutional Neural Networks for Automatic Meter Reading

A dataset containing 5000 images with 37,993 thousand relationships. The dataset contains 100 object categories and 70 predicate categories connecting those objects together. Source: Visual Relationship Detection with Language prior and Softmax

TDW is a 3D virtual world simulation platform, utilizing state-of-the-art video game engine technology. A TDW simulation consists of two components: a) the Build, a compiled executable running on the Unity3D Engine, which is responsible for image rendering, audio synthesis and physics simulations; and b) the Controller, an external Python interface to communicate with the build.

The Apron Dataset focuses on training and evaluating classification and detection models for airport-apron logistics. In addition to bounding boxes and object categories the dataset is enriched with meta parameters to quantify the models’ robustness against environmental influences.

This dataset contains 12,500 meter images acquired in the field by the employees of the Energy Company of Paraná (Copel), which directly serves more than 4 million consuming units, across 395 cities and 1,113 locations (i.e., districts, villages and settlements), located in the Brazilian state of Paraná. Copel-AMR is composed of images captured in unconstrained scenarios, which typically include blur (due to camera motion), dirt, scale variations, in-plane and out-of-plane rotations, reflections, shadows, and occlusions. In 2,500 images (i.e., 20% of the dataset), it is not even possible to perform the meter reading due to occlusions or faulty meters. The images have a resolution of 480×640 or 640×480 pixels, depending on the orientation in which they were taken. Considering that the meter is operational and that there are no occlusions, these resolutions are enough for the meter reading to be legible. The dataset was randomly split as follows: 5,000 images for training, 5,000 images for testing and 2,500 images for validation, following the split protocol (i.e., 40%/40%/20%) used in the UFPR-AMR dataset. For reproducibility purposes, the subsets generated are explicitly available along with the Copel-AMR dataset. For each image in our dataset, we manually labeled the meter reading, the position (x, y) of each of the four corners of the counter, and a bounding box (x, y, w, h) for each digit. Corner annotations – which can be converted to a bounding box – enable the counter to be rectified, while bounding boxes enable the training of object detectors. Source: Towards Image-based Automatic Meter Reading in Unconstrained Scenarios: A Robust and Efficient Approach