Video Segmentation

Datasets drive vision progress, yet existing driving datasets are impoverished in terms of visual content and supported tasks to study multitask learning for autonomous driving. Researchers are usually constrained to study a small set of problems on one dataset, while real-world computer vision applications require performing tasks of various complexities. We construct BDD100K, the largest driving video dataset with 100K videos and 10 tasks to evaluate the exciting progress of image recognition algorithms on autonomous driving. The dataset possesses geographic, environmental, and weather diversity, which is useful for training models that are less likely to be surprised by new conditions. Based on this diverse dataset, we build a benchmark for heterogeneous multitask learning and study how to solve the tasks together. Our experiments show that special training strategies are needed for existing models to perform such heterogeneous tasks. BDD100K opens the door for future studies in this important venue. More detail is at the dataset home page.

SegTrack v2 is a video segmentation dataset with full pixel-level annotations on multiple objects at each frame within each video. Source: https://web.engr.oregonstate.edu/~lif/SegTrack2/dataset.html Image Source: https://www.researchgate.net/publication/325842926_Semantic_Video_Segmentation_A_Review_on_Recent_Approaches

A novel large-scale corpus of manual annotations for the SoccerNet video dataset, along with open challenges to encourage more research in soccer understanding and broadcast production. Source: SoccerNet-v2 : A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos

BURST is a benchmark suite built upon TAO that requires tracking and segmenting multiple objects from camera video. The benchmark contains 6 different sub-tasks divided into 2 groups that all share the same data for training/validation/testing. ##### Class-guided 1. Common: Track and segment all objects belonging to a set of 78 common classes (based on the COCO class set) 2. Long-tail: Track and segment all objects belonging to an extended set of 482 object classes (based on the LVIS class set) 3. Open-world: Methods are only allowed to use the annotations of the 78 common classes during training, but during inference they are expected to track and segment all 482 object classes (class label predictions are not required) ##### Exemplar-guided 4. Mask: Track and segment all objects in the video for which the first-frame object masks are given. This task is identical to Video Object Segmentation (VOS). 5. Box: Track and segment all objects in the video for which the first-frame object bounding-boxes are given. 6. Point: Track and segment all objects in the video for which we are only given the (x,y) point coordinates of the mask centroid in the first-frame in which the objects appear. An illustration of the task hierarchy is given here and a detailed explanation is given in Sec. 5 of the dataset paper

Dynamic Replica is a synthetic dataset of stereo videos featuring humans and animals in virtual environments. It is a benchmark for dynamic disparity/depth estimation and 3D reconstruction consisting of 145,200 stereo frames (524 videos). The dataset contains annotations for left and right views that include: camera intrinsics and extrinsics, image depth, instance segmentation masks, binary foreground / background segmentation masks, optical flow, long-range pixel trajectories

EgoProceL is a large-scale dataset for procedure learning. It consists of 62 hours of egocentric videos recorded by 130 subjects performing 16 tasks for procedure learning. EgoProceL contains videos and key-step annotations for multiple tasks from CMU-MMAC, EGTEA Gaze+, and individual tasks like toy-bike assembly, tent assembly, PC assembly, and PC disassembly. EgoProceL overcomes the limitations of third-person videos. As, using third-person videos makes the manipulated object small in appearance and often occluded by the actor, leading to significant errors. In contrast, we observe that videos obtained from first-person (egocentric) wearable cameras provide an unobstructed and clear view of the action.

We learn high fidelity human depths by leveraging a collection of social media dance videos scraped from the TikTok mobile social networking application. It is by far one of the most popular video sharing applications across generations, which include short videos (10-15 seconds) of diverse dance challenges as shown above. We manually find more than 300 dance videos that capture a single person performing dance moves from TikTok dance challenge compilations for each month, variety, type of dances, which are moderate movements that do not generate excessive motion blur. For each video, we extract RGB images at 30 frame per second, resulting in more than 100K images. We segmented these images using Removebg application, and computed the UV coordinates from DensePose. Download TikTok Dataset: * Please use the dataset only for the research purpose. * The dataset can be viewed and downloaded from the Kaggle page. (you need to make an account in Kaggle to be able to download the data. It is free!) * The dataset can also be downloaded from here (42 GB). The dataset resolution is: (1080 x 604) * The original YouTube videos corresponding to each sequence and the dance name can be downloaded from here (2.6 GB).

ClipShots is a large-scale dataset for shot boundary detection collected from Youtube and Weibo covering more than 20 categories, including sports, TV shows, animals, etc. In contrast to previous shot boundary detection datasets, e.g. TRECVID and RAI, which only consist of documentaries or talk shows where the frames are relatively static, ClipShots contains moslty short videos from Youtube and Weibo. Many short videos are home-made, with more challenges, e.g. hand-held vibrations and large occlusion. The types of these videos are various, including movie spotlights, competition highlights, family videos recorded by mobile phones etc. Each video has a length of 1-20 minutes. The gradual transitions in the dataset include dissolve, fade in fade out, and sliding in sliding out. Source: https://github.com/Tangshitao/ClipShots

TRECVID is a yearly set of competitions centered on video retrieval and indexing, hosting a variety of video data sets. Source: YouTube-BoundingBoxes: A Large High-PrecisionHuman-Annotated Data Set for Object Detection in Video Image Source: https://www-nlpir.nist.gov/projects/tv2016/tv2016.html

This is a dataset for a shot boundary detection task. The dataset contains 2 existing datasets and 19 manually marked up open source videos with a total length of more than 1200 minutes and 10000 scene transitions. The dataset includes different types of videos with different resolutions from 360×288 to 1920×1080 in MP4 and MKV formats. Videos include samples in RGB scale or in grayscale with FPS from 23 to 60.

Operating rooms (ORs) are complex, high-stakes environments requiring precise understanding of interactions among medical staff, tools, and equipment for enhancing surgical assistance, situational awareness, and patient safety. Current datasets fall short in scale, realism and do not capture the multimodal nature of OR scenes, limiting progress in OR modeling. To this end, we introduce MM-OR, a realistic and large-scale multimodal spatiotemporal OR dataset, and the first dataset to enable multimodal scene graph generation. MM-OR captures comprehensive OR scenes containing RGB-D data, detail views, audio, speech transcripts, robotic logs, and tracking data and is annotated with panoptic segmentations, semantic scene graphs, and downstream task labels. Further, we propose MM2SG, the first multimodal large vision-language model for scene graph generation, and through extensive experiments, demonstrate its ability to effectively leverage multimodal inputs. Together, MM-OR and MM2SG establish a new benchmark for holistic OR understanding, and open the path towards multimodal scene analysis in complex, high-stakes environments. Paper: https://arxiv.org/abs/2503.02579

A dataset dedicated to multi-object, multi-actor activity parsing. The dataset contains * Video-level labels (activities) * Segment-level labels (sub-activities) * Atomic actions (spatio-temporal scene graph) The scene graph annotations contain object/actor classes and bounding boxes, relationship annotations, and object/actor attributes.

PETRAW data set was composed of 150 sequences of peg transfer training sessions. The objective of the peg transfer session is to transfer 6 blocks from the left to the right and back. Each block must be extracted from a peg with one hand, transferred to the other hand, and inserted in a peg at the other side of the board. All cases were acquired by a non-medical expert on the LTSI Laboratory from the University of Rennes. The data set was divided into a training data set composed of 90 cases and a test data set composed of 60 cases. A case was composed of kinematic data, a video, semantic segmentation of each frame, and workflow annotation.

A large-scale video portrait dataset that contains 291 videos from 23 conference scenes with 14K frames. This dataset contains various teleconferencing scenes, various actions of the participants, interference of passers-by and illumination change.

This is a video and image segmentation dataset for human head and shoulders, relevant for creating elegant media for videoconferencing and virtual reality applications. The source data includes ten online conference-style green screen videos. The authors extracted 3600 frames from the videos and generated the ground truth masks for each character in the video, and then applied virtual background to the frames to generate the training/testing sets.

A benchmark that focuses on the sampling dilemma in long-video tasks. The LSDBench dataset is designed to evaluate the sampling efficiency of long-video VLMs. It consists of multiple-choice question-answer pairs based on hour-long videos, focusing on dense and short-duration actions with high Necessary Sampling Density (NSD).

Infinity AI's Spills Basic Dataset is a synthetic, open-source dataset for safety applications. It features 150 videos of photorealistic liquid spills across 15 common settings. Spills take on in-context reflections, caustics, and depth based on the surrounding environment, lighting, and floor. Each video contains a spill of unique properties (size, color, profile, and more) and is accompanied by pixel-perfect labels and annotations. This dataset can be used to develop computer vision algorithms to detect the location and type of spill from the perspective of a fixed camera. ## Key Features + 150 videos + 4 environments where spills commonly occur + 15 unique indoor scenes + Realistic spill appearance + Diverse variation in spill geometry + Multiple spill colors + Rich annotations ## Annotations Each video is accompanied by a rich set of pixel-perfect labels and annotations, including: + Spill characteristics (size, color, profile, depth, etc.) + Segmentation masks + Bounding boxes For the full description of labels and metadata, check out the README. ## File Size Total dataset size: 890MB All 150 videos are contained in a single zipped folder (560MB). Video resolutions (all mp4): 960 x 720 at 24fps 960 x 640 at 24fps 960 x 637 at 24fps 960 x 384 at 24fps. ## Resources + Github README: full dataset and annotation descriptions + Demo Jupyter notebook: Shows how to (1) filter/query the dataset based on metadata, and (2) how to visualize labels + Questions? We’re happy to chat asynchronously via email or hop on a call. Just send us a note at info@toinfinity.ai (this goes to all of the Infinity AI founders). ## Terms and Conditions This work is licensed under a Creative Commons Attribution 4.0 International License. Both academic and commercial applications are allowed. ## Build better models faster with synthetic data Infinity AI, the creator of this dataset, is a synthetic data company. You can download all of Infinity's synthetic datasets and APIs at the Infinity Marketplace. Feel free to get in touch in you want custom data, or want to book a scoping session on how to make synthetic data your competitive edge. www.infinity.ai | LinkedIn | Synthetic Data Blog