Event-based vision

The Event-Camera Dataset is a collection of datasets with an event-based camera for high-speed robotics. The data also include intensity images, inertial measurements, and ground truth from a motion-capture system. An event-based camera is a revolutionary vision sensor with three key advantages: a measurement rate that is almost 1 million times faster than standard cameras, a latency of 1 microsecond, and a high dynamic range of 130 decibels (standard cameras only have 60 dB). These properties enable the design of a new class of algorithms for high-speed robotics, where standard cameras suffer from motion blur and high latency. All the data are released both as text files and binary (i.e., rosbag) files. Source: The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM

The Multi Vehicle Stereo Event Camera (MVSEC) dataset is a collection of data designed for the development of novel 3D perception algorithms for event based cameras. Stereo event data is collected from car, motorbike, hexacopter and handheld data, and fused with lidar, IMU, motion capture and GPS to provide ground truth pose and depth images. Source: EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras

DSEC is a stereo camera dataset in driving scenarios that contains data from two monochrome event cameras and two global shutter color cameras in favorable and challenging illumination conditions. In addition, we collect Lidar data and RTK GPS measurements, both hardware synchronized with all camera data. One of the distinctive features of this dataset is the inclusion of VGA-resolution event cameras. Event cameras have received increasing attention for their high temporal resolution and high dynamic range performance. However, due to their novelty, event camera datasets in driving scenarios are rare. This work presents the first high-resolution, large-scale stereo dataset with event cameras.

TUM-VIE is an event camera dataset for developing 3D perception and navigation algorithms. It contains handheld and head-mounted sequences in indoor and outdoor environments with rapid motion during sports and high dynamic range. TUM-VIE includes challenging sequences where state-of-the art VIO fails or results in large drift. Hence, it can help to push the boundary on event-based visual-inertial algorithms.

Contains 50 minutes of footage with both color frames and events. CED features a wide variety of indoor and outdoor scenes. Source: CED: Color Event Camera Dataset

The N-ImageNet dataset is an event-camera counterpart for the ImageNet dataset. The dataset is obtained by moving an event camera around a monitor displaying images from ImageNet. N-ImageNet contains approximately 1,300k training samples and 50k validation samples. In addition, the dataset also contains variants of the validation dataset recorded under a wide range of lighting or camera trajectories. Additional details about the dataset are explained in the paper available through this link. Please cite this paper if you make use of the dataset.

Large-scale single-object tracking dataset, containing 108 sequences with a total length of 1.5 hours. FE108 provides ground truth annotations on both the frame and event domain. The annotation frequency is up to 40Hz and 240Hz for the frame and event domains, respectively. FE108 is the largest event-frame-based dataset for single object tracking, and also offers the highest annotation frequency in the event domain.

In this work, we propose a general dataset for Color-Event camera based Single Object Tracking, termed COESOT. It contains 1354 color-event videos with 478,721 RGB frames. We split them into a training and testing subset, which contains 827 and 527 videos, respectively. The videos are collected from both outdoor and indoor scenarios (such as the street, zoo, and home) using the DAVIS346 event camera with a zoom lens. Therefore, our videos can reflect the variation in the distance at depth, but other datasets are failed to. Different from existing benchmarks which contain limited categories, our proposed COESOT covers a wider range of object categories (90 classes), as shown in Fig. 3 (a). It mainly reflects four groups, including persons, animals, electronics, and other goods. The ground truth of the proposed COESOT dataset is densely annotated, i.e., in a frame-by frame way. The absent label is also provided to help researchers design their trackers. Inspired by VisEvent [44], we annotate each testing video sequence with 17 attributes to help researchers evaluate their trackers in specific challenging environments, e.g., full occlusion (FOC), deformation (DEF), rotation (ROT), fast motion (FM), partially occlusion (POC), low illumination (LI), scale variation (SV), background object motion (BOM), motion blur (MB), overexposure (OE), etc. The distribution of videos in each attribute is shown in Fig. 3 (b). The statistical distribution of the ground truth center position is shown in Fig 3 (c). More details can be found in our supplementary materials.

Event-Human3.6m is a challenging dataset for event-based human pose estimation by simulating events from the RGB Human3.6m dataset. It is built by converting the RGB recordings of Human3.6m into events and synchronising raw joints ground-truth with events frames through interpolation.

Used to show systematic performance improvement in applications such as high frame-rate video synthesis, feature/corner detection and tracking, as well as high dynamic range image reconstruction. Source: Joint Filtering of Intensity Images and Neuromorphic Events for High-Resolution Noise-Robust Imaging

We use Kubric and ESIM simulator to make our EKubric dataset, which has 15,367 RGB-PointCloud-Event pairs with annotations (including optical flow, scene flow, surface normal, semantic segmentation and object coordinates ground truths).

This synthetic event dataset is used in Robust e-NeRF to study the collective effect of camera speed profile, contrast threshold variation and refractory period on the quality of NeRF reconstruction from a moving event camera. It is simulated using an improved version of ESIM with three different camera configurations of increasing difficulty levels (i.e. easy, medium and hard) on seven Realistic Synthetic $360^{\circ}$ scenes (adopted in the synthetic experiments of NeRF), resulting in a total of 21 sequence recordings. Please refer to the Robust e-NeRF paper for more details. The dataset allows for a retrospective comparison between event-based and image-based NeRF reconstruction methods, as the event sequences were simulated under highly similar conditions as the NeRF synthetic dataset. In particular, we adopt the same camera intrinsics and camera distance to the object at the origin. Furthermore, the event camera travels in a hemi-/spherical spiral motion about the object, thereby having a similar camera pose distribution for training. Apart from that, we also use the same test camera poses/views. Nonetheless, this new synthetic event dataset is not only specific to NeRF reconstruction, but also suitable for novel view synthesis, 3D reconstruction, localization and SLAM in general.

Overview The Spike-X4K Dataset is a high-resolution image reconstruction resource tailored for the latest advancements in spike camera technology. It is designed to meet the demands of modern spike cameras with a resolution of 1000×1000 pixels, surpassing the capabilities of previous datasets like spike-REDS, which was limited to a resolution of 250×400 pixels. ## Dataset Characteristics - Resolution: 1000×1000 pixels, aligning with state-of-the-art spike camera imaging standards. - Temporal Depth: The dataset captures the temporal dynamics of scenes with high-speed motion, providing a temporal sequence of spike frames. - Content: It includes both synthetic and real-world datasets, offering a diverse range of high-speed motion scenarios for training and testing image reconstruction models. - Pairs: The dataset comprises 1200 spike stream-ground truth image pairs for training and 45 pairs for testing, ensuring robust model evaluation. ## Motivation and Content Summary The development of the Spike-X4K Dataset was motivated by the need for a more representative and higher resolution dataset that could effectively train and evaluate image reconstruction models for spike cameras. Spike cameras, with their unique ability to capture photons independently at each pixel and generate binary spike streams, offer high temporal resolution and low latency, which are crucial for high-speed imaging. However, converting these spike streams into high-quality images requires sophisticated algorithms, and the Spike-X4K Dataset provides the necessary data to develop and refine these algorithms. ## Potential Use Cases 1. Algorithm Development: For researchers and engineers working on spike image reconstruction algorithms specific to spike camera data. 2. Benchmarking: As a standard for benchmarking the performance of various image reconstruction models against state-of-the-art techniques. 3. Training and Testing: Providing a large and diverse set of data for training deep learning models to handle high-speed motion imaging. 4. Feature Extraction: Enabling the study and improvement of feature extraction techniques from spike streams, including spatial and temporal features. 5. Spike Image Reconstruction Tasks: Supporting a wide range of computer vision tasks that can benefit from high-resolution, high-speed imaging, such as motion analysis, object tracking, and event detection.

The MS-EVS Dataset is the first large-scale event-based dataset for face detection. - Regular images (APS) and simulated events (EVS) - Multispectral data (RGB and/or near IR) - Very diverse faces and data quality (for robust face detection) It is divided in four subsets: - N-YoutubeFaces, 3k+ celebrity faces from Youtube videos, in color (simulated events). - N-MobiFace, 80 unedited mobile live color videos (simulated events). - N-SpectralFace, 69 multispectral videos captured in-house (simulated events). - Real-SpectralFace, a few sequences with real multispectral events (Grayscale and Infrared). | Dataset | APS | Simulated EVS | Real EVS | Data | |:---------------------:|:-------:|:-----------------:|:------------:|:--------------------------------------:| | N-YoutubeFaces | x | x | | Color (RGB) | | N-MobiFace | x | x | | Color (RGB) | | N-SpectralFace | x | x | | Multispectral (9 visible + 1 infrared) | | Real-SpectralFace | x | | x | Multispectral (1 visible + 1 infrared) |