Unsupervised Video Object Segmentation

DAVIS17 is a dataset for video object segmentation. It contains a total of 150 videos - 60 for training, 30 for validation, 60 for testing Source: Siam R-CNN: Visual Tracking by Re-Detection Image Source: https://www.researchgate.net/figure/LucidTracker-qualitative-results-on-DAVIS-17-test-dev-set-Frames-sampled-along-the_fig5_331792902

DAVIS16 is a dataset for video object segmentation which consists of 50 videos in total (30 videos for training and 20 for testing). Per-frame pixel-wise annotations are offered. Source: Learning Discriminative Feature with CRF for Unsupervised Video Object Segmentation Image Source: https://davischallenge.org/

The Freiburg-Berkeley Motion Segmentation Dataset (FBMS-59) is an extension of the BMS dataset with 33 additional video sequences. A total of 720 frames is annotated. It has pixel-accurate segmentation annotations of moving objects. FBMS-59 comes with a split into a training set and a test set. Source: https://lmb.informatik.uni-freiburg.de/resources/datasets/ Image Source: https://lmb.informatik.uni-freiburg.de/resources/datasets/

SegTrack v2 is a video segmentation dataset with full pixel-level annotations on multiple objects at each frame within each video. Source: https://web.engr.oregonstate.edu/~lif/SegTrack2/dataset.html Image Source: https://www.researchgate.net/publication/325842926_Semantic_Video_Segmentation_A_Review_on_Recent_Approaches

Our task is to localize and provide a pixel-level mask of an object on all video frames given a language referring expression obtained either by looking at the first frame only or the full video. To validate our approach we employ two popular video object segmentation datasets, DAVIS16 [38] and DAVIS17 [42]. These two datasets introduce various challenges, containing videos with single or multiple salient objects, crowded scenes, similar looking instances, occlusions, camera view changes, fast motion, etc. DAVIS16 [38] consists of 30 training and 20 test videos of diverse object categories with all frames annotated with pixel-level accuracy. Note that in this dataset only a single object is annotated per video. For the multiple object video segmentation task we consider DAVIS17. Compared to DAVIS16, this is a more challenging dataset, with multiple objects annotated per video and more complex scenes with more distractors, occlusions, smaller objects, and fine structures. Overall, DAVIS17 consists of a training set with 60 videos, and a validation/test-dev/test-challenge set with 30 sequences each. As our goal is to segment objects in videos using language specifications, we augment all objects annotated with mask labels in DAVIS16 and DAVIS17 with non-ambiguous referring expressions. We follow the work of [34] and ask the annotator to provide a language description of the object, which has a mask annotation, by looking only at the first frame of the video. Then another annotator is given the first frame and the corresponding description, and asked to identify the referred object. If the annotator is unable to correctly identify the object, the description is corrected to remove ambiguity and to specify the object uniquely. We have collected two referring expressions per target object annotated by non-computer vision experts (Annotator 1, 2). However, by looking only at the 1st frame, the obtained referring expressions may potentially be invalid for an entire video. (We actually quantified that only∼ 15% of the collected descriptions become invalid over time and it does not affect strongly segmentation results as temporal consistency step helps to disambiguate some of such cases, see the supp. material for details.) Besides, in many applications, such as video editing or video-based advertisement, the user has access to a full video. Providing a language query which is valid for all frames might decrease the editing time and result in more coherent predictions. Thus, on DAVIS17 we asked the workers to provide a description of the object by looking at the full video. We have collected one expression of the full video type per target object. Future work may choose to use either setting. The average length for the first frame/full video expressions is 5.5/6.3 words. For DAVIS17 first frame annotations we notice that descriptions given by Annotator 1 are longer than the ones by Annotator 2 (6.4 vs. 4.6 words). We evaluate the effect of description length on the grounding performance in §5. Besides, the expressions relevant to a full video mention verbs more often than the first frame descriptions (44% vs. 25%). This is intuitive, as referring to an object which changes its appearance and position over time may require mentioning its actions. Adjectives are present in over 50% for all annotations. Most of them refer to colors (over 70%), shapes and sizes (7%) and spatial/ordering words (6% first frame vs. 13% full video expressions). The full video expressions also have a higher number of adverbs and prepositions, and overall are more complex than the ones provided for the first frame. Overall augmented DAVIS16/17 contains ∼ 1.2k referring expressions for more than 400 objects on 150 videos with ∼ 10k frames. We believe the collected data will be of interest to segmentation as well as vision and language communities, providing an opportunity to explore language as alternative input for video object segmentation.

CoMplex video Object SEgmentation (MOSE) is a dataset to study the tracking and segmenting objects in complex environments. MOSE contains 2,149 video clips and 5,200 objects from 36 categories, with 431,725 high-quality object segmentation masks. The most notable feature of MOSE dataset is complex scenes with crowded and occluded objects. Source: MOSE: A New Dataset for Video Object Segmentation in Complex Scenes

The Freiburg-Berkeley Motion Segmentation Dataset (FBMS-59) is a dataset for motion segmentation, which extends the BMS-26 dataset with 33 additional video sequences. A total of 720 frames is annotated. FBMS-59 comes with a split into a training set and a test set. Typical challenges appear in both sets. Source: https://lmb.informatik.uni-freiburg.de/resources/datasets/moseg.en.html Image Source: https://lmb.informatik.uni-freiburg.de/resources/datasets/moseg.en.html

BL30K is a synthetic dataset rendered using Blender with ShapeNet's data. We break the dataset into six segments, each with approximately 5K videos. The videos are organized in a similar format as DAVIS and YouTubeVOS, so dataloaders for those datasets can be used directly. Each video is 160 frames long, and each frame has a resolution of 768*512. There are 3-5 objects per video, and each object has a random smooth trajectory -- we tried to optimize the trajectories in a greedy fashion to minimize object intersection (not guaranteed), with occlusions still possible (happen a lot in reality). See MiVOS for details.