Multiple Object Tracking

KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) is one of the most popular datasets for use in mobile robotics and autonomous driving. It consists of hours of traffic scenarios recorded with a variety of sensor modalities, including high-resolution RGB, grayscale stereo cameras, and a 3D laser scanner. Despite its popularity, the dataset itself does not contain ground truth for semantic segmentation. However, various researchers have manually annotated parts of the dataset to fit their necessities. Álvarez et al. generated ground truth for 323 images from the road detection challenge with three classes: road, vertical, and sky. Zhang et al. annotated 252 (140 for training and 112 for testing) acquisitions – RGB and Velodyne scans – from the tracking challenge for ten object categories: building, sky, road, vegetation, sidewalk, car, pedestrian, cyclist, sign/pole, and fence. Ros et al. labeled 170 training images and 46 testing images (from the visual odometry challenge) with 11 classes: building, tree, sky, car, sign, road, pedestrian, fence, pole, sidewalk, and bicyclist. Source: A Review on Deep Learning Techniques Applied to Semantic Segmentation Image Source: http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d

The Waymo Open Dataset is comprised of high resolution sensor data collected by autonomous vehicles operated by the Waymo Driver in a wide variety of conditions. The Waymo Open Dataset currently contains 1,950 segments. The authors plan to grow this dataset in the future. Currently the datasets includes: * 1,950 segments of 20s each, collected at 10Hz (390,000 frames) in diverse geographies and conditions * Sensor data * 1 mid-range lidar * 4 short-range lidars * 5 cameras (front and sides) * Synchronized lidar and camera data * Lidar to camera projections * Sensor calibrations and vehicle poses * Labeled data * Labels for 4 object classes - Vehicles, Pedestrians, Cyclists, Signs * High-quality labels for lidar data in 1,200 segments * 12.6M 3D bounding box labels with tracking IDs on lidar data * High-quality labels for camera data in 1,000 segments * 11.8M 2D bounding box labels with tracking IDs on camera data

Datasets drive vision progress, yet existing driving datasets are impoverished in terms of visual content and supported tasks to study multitask learning for autonomous driving. Researchers are usually constrained to study a small set of problems on one dataset, while real-world computer vision applications require performing tasks of various complexities. We construct BDD100K, the largest driving video dataset with 100K videos and 10 tasks to evaluate the exciting progress of image recognition algorithms on autonomous driving. The dataset possesses geographic, environmental, and weather diversity, which is useful for training models that are less likely to be surprised by new conditions. Based on this diverse dataset, we build a benchmark for heterogeneous multitask learning and study how to solve the tasks together. Our experiments show that special training strategies are needed for existing models to perform such heterogeneous tasks. BDD100K opens the door for future studies in this important venue. More detail is at the dataset home page.

The MOTChallenge datasets are designed for the task of multiple object tracking. There are several variants of the dataset released each year, such as MOT15, MOT17, MOT20. Source: MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking

Consists of 100 challenging video sequences captured from real-world traffic scenes (over 140,000 frames with rich annotations, including occlusion, weather, vehicle category, truncation, and vehicle bounding boxes) for object detection, object tracking and MOT system. Source: UA-DETRAC: A New Benchmark and Protocol for Multi-Object Detection and Tracking Image Source: UA-DETRAC: A New Benchmark and Protocol for Multi-Object Detection and Tracking

MOT20 is a dataset for multiple object tracking. The dataset contains 8 challenging video sequences (4 train, 4 test) in unconstrained environments, from crowded places such as train stations, town squares and a sports stadium. Image Source: https://motchallenge.net/vis/MOT20-04

Motivation Multi-object tracking (MOT) is a fundamental task in computer vision, aiming to estimate objects (e.g., pedestrians and vehicles) bounding boxes and identities in video sequences. Prevailing human-tracking MOT datasets mainly focus on pedestrians in crowded street scenes (e.g., MOT17/20) or dancers in static scenes (DanceTrack). In spite of the increasing demands for sports analysis, there is a lack of multi-object tracking datasets for a variety of sports scenes, where the background is complicated, players possess rapid motion and the camera lens moves fast. To this purpose, we propose a large-scale multi-object tracking dataset named SportsMOT, consisting of 240 video clips from 3 categories (i.e., basketball, football and volleyball). The objective is to only track players on the playground (i.e., except for a number of spectators, referees and coaches) in various sports scenes. We expect SportsMOT to encourage the community to concentrate more on the complicated sports scenes. ## Characteristics - Large scale - Fine Annotations - Player id consistency - No shot change - High and fixed resolution(1080P) - ... ## Focus - Diverse sports scenes - Complex motion patterns - Challenging re-id ## Download ### Examples You can download the example for SportsMOT. - OneDrive - Baidu Netdisk, password: 4dnw ### Official Dataset Please Sign up in codalab, and participate in our competition. Download links are available in Participate/Get Data. ## News - SportsMOT is used for DeeperAction@ECCV-2022. - Refer to github repo: MCG-NJU/SportsMOT for the latest info.

The Multi-Object and Segmentation (MOTS) benchmark [2] consists of 21 training sequences and 29 test sequences. It is based on the KITTI Tracking Evaluation 2012 and extends the annotations to the Multi-Object and Segmentation (MOTS) task. To this end, we added dense pixel-wise segmentation labels for every object. We evaluate submitted results using the metrics HOTA, CLEAR MOT, and MT/PT/ML. We rank methods by HOTA [1]. Our development kit and GitHub evaluation code provide details about the data format as well as utility functions for reading and writing the label files. (adapted for the segmentation case). Evaluation is performed using the code from the TrackEval repository. [1] J. Luiten, A. Os̆ep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixé, B. Leibe: HOTA: A Higher Order Metric for Evaluating Multi-object Tracking. IJCV 2020. [2] P. Voigtlaender, M. Krause, A. Os̆ep, J. Luiten, B. Sekar, A. Geiger, B. Leibe: MOTS: Multi-Object Tracking and Segmentation. CVPR 2019.

RADIATE (RAdar Dataset In Adverse weaThEr) is new automotive dataset created by Heriot-Watt University which includes Radar, Lidar, Stereo Camera and GPS/IMU. The data is collected in different weather scenarios (sunny, overcast, night, fog, rain and snow) to help the research community to develop new methods of vehicle perception. The radar images are annotated in 7 different scenarios: Sunny (Parked), Sunny/Overcast (Urban), Overcast (Motorway), Night (Motorway), Rain (Suburban), Fog (Suburban) and Snow (Suburban). The dataset contains 8 different types of objects (car, van, truck, bus, motorbike, bicycle, pedestrian and group of pedestrians). Source: https://github.com/marcelsheeny/radiate_sdk Image Source: https://github.com/marcelsheeny/radiate_sdk

A new large-scale dataset for understanding human motions, poses, and actions in a variety of realistic events, especially crowd & complex events. It contains a record number of poses (>1M), the largest number of action labels (>56k) for complex events, and one of the largest number of trajectories lasting for long terms (with average trajectory length >480). Besides, an online evaluation server is built for researchers to evaluate their approaches. Source: Human in Events: A Large-Scale Benchmark for Human-centric Video Analysis in Complex Events

GMOT-40 is the first public dense dataset for Generic Multiple Object Tracking (GMOT). It contains 40 carefully annotated sequences evenly distributed among 10 object categories. Beyond the data, a challenging protocal, one-shot GMOT, is adopted and a series of baseline algorithms is introduced. GMOT-40 is featured in Dense objects. Over 80 objects of the same class could appear in one frame. High-quality label. Manual annotation with careful inspection in each frame. Diversity in target. Large variability between classes and within the sequences of the same class. Real world challenges. Occlusion, target enter/exiting, motion blur, deformation and so on. Challenging protocol. One-shot GMOT protocol is adopted for evaluation. New baselines. A series of baseline methods for one-shot GMOT task is introduced.

PathTrack is a dataset for person tracking which contains more than 15,000 person trajectories in 720 sequences. Source: PathTrack: Fast Trajectory Annotation with Path Supervision

Labeled Pedestrian in the Wild (LPW) is a pedestrian detection dataset that contains 2,731 pedestrians in three different scenes where each annotated identity is captured by from 2 to 4 cameras. The LPW features a notable scale of 7,694 tracklets with over 590,000 images as well as the cleanliness of its tracklets. It distinguishes from existing datasets in three aspects: large scale with cleanliness, automatically detected bounding boxes and far more crowded scenes with greater age span. This dataset provides a more realistic and challenging benchmark, which facilitates the further exploration of more powerful algorithms.

Multi-camera Multiple People Tracking (MMPTRACK) dataset has about 9.6 hours of videos, with over half a million frame-wise annotations. The dataset is densely annotated, e.g., per-frame bounding boxes and person identities are available, as well as camera calibration parameters. Our dataset is recorded with 15 frames per second (FPS) in five diverse and challenging environment settings., e.g., retail, lobby, industry, cafe, and office. This is by far the largest publicly available multi-camera multiple people tracking dataset. We expect the availability of such large-scale multi-camera multiple people tracking dataset will encourage more participants in this research topic. This dataset is also valuable for the evaluation of other tasks, such as multi-view people detection and monocular multiple people tracking.

Caltech Fish Counting Dataset (CFC) is a large-scale dataset for detecting, tracking, and counting fish in sonar videos. This dataset contains over 1,500 videos sourced from seven different sonar cameras.

PersonPath22 is a large-scale multi-person tracking dataset containing 236 videos captured mostly from static-mounted cameras, collected from sources where we were given the rights to redistribute the content and participants have given explicit consent. Each video has ground-truth annotations including both bounding boxes and tracklet-ids for all the persons in each frame.

The Omni-MOT is realistic CARLA based large-scale dataset with over 14M frames for multiple vehicle tracking . The dataset comprises 14M+ frames, 250K tracks, 110 million bounding boxes, three weather conditions, three crowd levels and three camera views in five simulated towns. Source: Simultaneous Detection and Tracking with Motion Modelling for Multiple Object Tracking

SOMPT22 is a multi-object tracking (MOT) benchmark focused on surveillance-style pedestrian tracking. * 22 long video sequences (static pole-mounted cameras, 6 – 8 m height) * ~51 k annotated frames with bounding boxes + unique track IDs * Outdoor scenes with illumination changes, partial occlusions and appearance similarity * Single class: person * Split files ready for training/validation and standard MOT evaluation tools SOMPT22 aims to complement generic MOTChallenge-style datasets by stressing long-term ID maintenance under sparse-to-medium crowd density instead of dense, short clips. Homepage → https://sompt22.github.io Download → Google Drive link in the homepage Citation → ```bibtex @misc{simsek2022sompt22, author = {Simsek, Fatih Emre and Cigla, Cevahir and Kayabol, Koray}, title = {SOMPT22: A Surveillance Oriented Multi-Pedestrian Tracking Dataset}, year = {2022}, eprint = {2208.02580}, archivePrefix = {arXiv}, primaryClass = {cs.CV} }

YouTube-Hands includes 240 videos which are annotated with hand trajectories.