Video Classification

The Kinetics dataset is a large-scale, high-quality dataset for human action recognition in videos. The dataset consists of around 500,000 video clips covering 600 human action classes with at least 600 video clips for each action class. Each video clip lasts around 10 seconds and is labeled with a single action class. The videos are collected from YouTube. Source: Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey

The Charades dataset is composed of 9,848 videos of daily indoors activities with an average length of 30 seconds, involving interactions with 46 objects classes in 15 types of indoor scenes and containing a vocabulary of 30 verbs leading to 157 action classes. Each video in this dataset is annotated by multiple free-text descriptions, action labels, action intervals and classes of interacting objects. 267 different users were presented with a sentence, which includes objects and actions from a fixed vocabulary, and they recorded a video acting out the sentence. In total, the dataset contains 66,500 temporal annotations for 157 action classes, 41,104 labels for 46 object classes, and 27,847 textual descriptions of the videos. In the standard split there are7,986 training video and 1,863 validation video. Source: Temporal Reasoning Graph for Activity Recognition

The 20BN-SOMETHING-SOMETHING V2 dataset is a large collection of labeled video clips that show humans performing pre-defined basic actions with everyday objects. The dataset was created by a large number of crowd workers. It allows machine learning models to develop fine-grained understanding of basic actions that occur in the physical world. It contains 220,847 videos, with 168,913 in the training set, 24,777 in the validation set and 27,157 in the test set. There are 174 labels. Source Image Source

The Breakfast Actions Dataset comprises of 10 actions related to breakfast preparation, performed by 52 different individuals in 18 different kitchens. The dataset is one of the largest fully annotated datasets available. The actions are recorded “in the wild” as opposed to a single controlled lab environment. It consists of over 77 hours of video recordings. Source: https://serre-lab.clps.brown.edu/resource/breakfast-actions-dataset/ Image Source: https://serre-lab.clps.brown.edu/resource/breakfast-actions-dataset/

The YouTube-8M dataset is a large scale video dataset, which includes more than 7 million videos with 4716 classes labeled by the annotation system. The dataset consists of three parts: training set, validate set, and test set. In the training set, each class contains at least 100 training videos. Features of these videos are extracted by the state-of-the-art popular pre-trained models and released for public use. Each video contains audio and visual modality. Based on the visual information, videos are divided into 24 topics, such as sports, game, arts & entertainment, etc Source: Audio-Visual Embedding for Cross-Modal Music Video Retrieval through Supervised Deep CCA

The 20BN-SOMETHING-SOMETHING dataset is a large collection of labeled video clips that show humans performing pre-defined basic actions with everyday objects. The dataset was created by a large number of crowd workers. It allows machine learning models to develop fine-grained understanding of basic actions that occur in the physical world. It contains 108,499 videos, with 86,017 in the training set, 11,522 in the validation set and 10,960 in the test set. There are 174 labels. ⚠️ Attention: This is the outdated V1 of the dataset. V2 is available here. Source: https://20bn.com/datasets/something-something/v1 Image Source: https://20bn.com/datasets/something-something/v1

The COIN dataset (a large-scale dataset for COmprehensive INstructional video analysis) consists of 11,827 videos related to 180 different tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. The videos are all collected from YouTube. The average length of a video is 2.36 minutes. Each video is labelled with 3.91 step segments, where each segment lasts 14.91 seconds on average. In total, the dataset contains videos of 476 hours, with 46,354 annotated segments. Source: COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis

Contains 68,536 activity instances in 68.8 hours of first and third-person video, making it one of the largest and most diverse egocentric datasets available. Charades-Ego furthermore shares activity classes, scripts, and methodology with the Charades dataset, that consist of additional 82.3 hours of third-person video with 66,500 activity instances. Source: Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos

DAiSEE is a multi-label video classification dataset comprising of 9,068 video snippets captured from 112 users for recognizing the user affective states of boredom, confusion, engagement, and frustration "in the wild". The dataset has four levels of labels namely - very low, low, high, and very high for each of the affective states, which are crowd annotated and correlated with a gold standard annotation created using a team of expert psychologists. Source: DAiSEE: Towards User Engagement Recognition in the Wild

Home Action Genome is a large-scale multi-view video database of indoor daily activities. Every activity is captured by synchronized multi-view cameras, including an egocentric view. There are 30 hours of vides with 70 classes of daily activities and 453 classes of atomic actions.

A dataset dedicated to multi-object, multi-actor activity parsing. The dataset contains * Video-level labels (activities) * Segment-level labels (sub-activities) * Atomic actions (spatio-temporal scene graph) The scene graph annotations contain object/actor classes and bounding boxes, relationship annotations, and object/actor attributes.

Whereas the action recognition community has focused mostly on detecting simple actions like clapping, walking or jogging, the detection of fights or in general aggressive behaviors has been comparatively less studied. Such capability may be extremely useful in some video surveillance scenarios like in prisons, psychiatric or elderly centers or even in camera phones. After an analysis of previous approaches we test the well-known Bag-of-Words framework used for action recognition in the specific problem of fight detection, along with two of the best action descriptors currently available: STIP and MoSIFT. For the purpose of evaluation and to foster research on violence detection in video we introduce a new video database containing 1000 sequences divided in two groups: fights and non-fights. Experiments on this database and another one with fights from action movies show that fights can be detected with near 90% accuracy.

MetaVD is a Meta Video Dataset for enhancing human action recognition datasets. It provides human-annotated relationship labels between action classes across human action recognition datasets. MetaVD is proposed in the following paper: Yuya Yoshikawa, Yutaro Shigeto, and Akikazu Takeuchi. "MetaVD: A Meta Video Dataset for enhancing human action recognition datasets." Computer Vision and Image Understanding 212 (2021): 103276. [link] MetaVD integrates the following datasets: UCF101, HMDB51, ActivityNet, STAIR Actions, Charades, Kinetics-700 This repository does NOT provide videos in the datasets. For information on how to download the videos, please refer to the website of each dataset.

A dataset of cartoon video clips. For each video clip, the presence or absence of each feature was marked by the annotators. Malicious: Video content which may not be suitable for viewing by toddlers and pre-schoolers includes a set of clearly defined, trivial and intu-itive features as well as some complex and subtle audio andvideo features. The former includes forms of obscenity and violence on a higher level e.g. nudity, gore, etc. while the latter includes elements such as fast repetitive motion, loud music, disgusting and scary characters, smashing people or things, forms of aggression, loud music, screaming or shout-ing, gunshots and explosions. Benign: Educational videos and videos of nursery rhymes are usually considered to be appropriate for the toddlers and pre-schoolers; in fact some experts recommend letting kids watch them for a limited number of hours.Benign videos are characterized by a slower tempo, softer music or sound effects, moderate-paced motions, and soft-toned conversations. Most importantly, benign video content should not contain any indicators of malicious content as discussed earlier.

This large collection of over 161,000 video-label pairs of video clips, shows humans drawing letters and digits in the air, and is used to evaluate a model’s ability to classify articulated motions correctly. Unlike existing video datasets, AirLetters’ accurate classification predictions rely on discerning motion patterns and integrating information presented by the video over time (i.e., over many frames of video). That study revealed that while trivial for humans, accurate representations of complex articulated motions remain an open problem for end-to-end learning for video understanding models.

We introduce a video dataset Bukva for Russian Dactyl Recognition task. Bukva dataset size is about 27 GB, and it contains 3757 RGB videos with more than 101 samples for each RSL alphabet sign, including dynamic ones. The dataset is divided into training set and test set by subject user_id. The training set includes 3097 videos, and the test set includes 660 videos. The total video recording time is ~4 hours. About 17% of the videos are recorded in HD format, and 70% of the videos are in FullHD resolution.

Dataset for multimodal skills assessment focusing on assessing piano player’s skill level. Annotations include player's skills level, and song difficulty level. Bounding box annotations around pianists' hands are also provided.

APPROVE consists of curated YouTube videos annotated with educational content. APPROVE consists of 193 hours of expert-annotated videos with 19 classes (7 literacy codes, 11 math, and background) and each video is associated with approximately 3 labels on average.

Trailers12k is a movie trailer dataset comprised of 12,000 titles associated to ten genres. It distinguishes from other datasets by its collection procedure aimed at providing a high-quality publicly available dataset.

First of its kind paired win-fail action understanding dataset with samples from the following domains: “General Stunts,” “Internet Wins-Fails,” “Trick Shots,” & “Party Games.” The task is to identify successful and failed attempts at various activities. Unlike existing action recognition datasets, intra-class variation is high making the task challenging, yet feasible.

This dataset defines a total of 11 crowd motion patterns and it is composed of over 6000 video sequences with an average length of 100 frames per sequence. This documentation presents how to download and process the Crowd-11 dataset. If you use this dataset, please cite our paper: Camille Dupont, Luis Tobias, and Bertrand Luvison. "Crowd-11: A Dataset for Fine Grained Crowd Behaviour Analysis." In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017. Since this dataset is a composition of web videos and already existing datasets, we ask you to download and accept licence of each source and dataset. The construction of the Crowd-11 dataset is composed of two steps: # Step 1: Retrieve videos of interest from the web and/or pre-existing datasets ## Retrieve the pre-existing datasets of interest The pre-existing datasets are: | DATASET NAME | url | $SOURCE_NAME | | ------------- |:-----------------------------------------------------------------:| -------------:| | UMN | http://mha.cs.umn.edu/proj_events.shtml#crowd | umn | | AGORASET |https://github.com/piallai/Agoraset | agoraset | | PETS | http://www.cvg.reading.ac.uk/PETS2009/a.html#s3 | pets | | HOCKEY FIGHT | http://visilab.etsii.uclm.es/personas/oscar/FightDetection/ | hockey | | MOVIES | http://visilab.etsii.uclm.es/personas/oscar/FightDetection/ | peliculas | | CUHK | http://www.ee.cuhk.edu.hk/~jshao/CUHKcrowd_files/cuhk_crowd_dataset.htm | cuhk | | WWW | http://www.ee.cuhk.edu.hk/~jshao/WWWCrowdDataset.html | www | | WORLDEXPO'10 CROWD COUNTING | http://www.ee.cuhk.edu.hk/~xgwang/expo.html | shanghai | | VIOLENT-FLOWS | http://www.openu.ac.il/home/hassner/data/violentflows/ | violent_flow | These datasets should be stored in their "existing_datasets/$SOURCE_NAME/" folder: . └── existing_datasets ├── agoraset ├── cuhk ├── hockey ├── peliculas │ ├── fights │ └── noFights ├── pets ├── shanghai ├── umn ├── violent_flow └── www ## Copy the videos of interest from the datasets of interest The list of the videos of interest is in existing_datasets_urls.csv. To extract them into the VOI folder, execute: python existing_datasets_gathering.py The VOI folder should have the following structure: . └── VOI ├── agoraset ├── cuhk ├── hockey ├── peliculas ├── pets ├── shanghai ├── umn ├── violent_flow └── www ## Download the videos of interest from the web The web sources are: | SOURCE NAME | url | $SOURCE_NAME | | ------------- |:----------------------------------:| -------------:| | YOUTUBE | https://www.youtube.com/ | youtube | | GETTYIMAGES | http://www.gettyimages.fr/ | gettyimages | | POND5 | https://www.pond5.com/ | pond5 | The list of the web urls to download is in web_urls.csv. The web_urls.csv file's structure is as follows : | $SOURCE NAME | URL | OUTPUT_NAME | TS_MULTIPLIER | | ------------- |:---------------------:| ------------:| --------------:| We do not provide the script to download them, but many tools exist to do it (pytube, urllib, etc...). Note: a few videos have a ts_multiplier field. These video are in slow motion and the ts_multiplier is provided to speed them up (cf. SETPTS option in avconv). The downloaded videos should be stored in their VOI/$SOURCE_NAME folder, which should now have the following structure: . └── VOI ├── agoraset ├── cuhk ├── gettyimages ├── hockey ├── peliculas ├── pets ├── pond5 ├── shanghai ├── umn ├── violent_flow ├── youtube └── www # Step 2: Processing original videos into the Crowd-11 dataset Once the VOI folder is complete, a preprocessing step is required in order to crop and trim the original videos into the Crowd-11 dataset. The preprocessing.csv file's structure is as follows : | Videoname | Label | Frame_start | Frame_end | Top_left | Top_right | Width | Height | $SOURCE_NAME | Scene_number | Crop_number | | ------------- |:---------:| ------------:| ----------:| ---------:| ----------:| -----:| ------:| ------------:| ------------:| -----------:| Installation: You need to have avconv installed: sudo apt-get install avconv Then, you need to install several python package. A virtualeenv installation is recommended: virtualenv -p python3 py source py/bin/activate pip install sk-video Execution (in the virtualenv): python script_formating.py

Laser powder bed fusion (LBPF) is the additive manufacturing (3D printing) process for metals. RAISE-LPBF is a large dataset on the effect of laser power and laser dot speed in 316L stainless steel bulk material. Both process parameters are independently sampled for each scan line from a continuous distribution, so interactions of different parameter choices can be investigated. Process monitoring comprises on-axis high-speed (20k FPS) video. The data can be used to derive statistical properties of LPBF, as well as to build anomaly detectors. RAISE-LPBF-Laser is the machine learning benchmark to reconstruct the laser parameters of the RAISE-LBPF dataset. Paper: https://doi.org/10.1016/j.addlet.2023.100161

The Storytelling Video Dataset is a high-quality, human-reviewed multimodal dataset featuring over 700 full-body video recordings of native Russian speakers. Each video is 10+ minutes long and includes synchronized speech, facial expressions, gestures, and emotional variation. The dataset is ideal for research and development in: 🗣️ Speech-to-text & voice modeling (Russian) 😃 Emotion & gesture recognition 🤖 Multimodal learning & LLM alignment 🧍 Avatar generation & digital human training Key Features: Full-body framing (waist-up or knee-up) Clear facial expressions and gestures High-quality speech audio in Russian Natural storytelling with emotional diversity Preview (10 Participants): We’ve prepared a short compilation of 10 different speakers from the dataset. Each clip is taken from a unique 10-minute unscripted video, featuring full-body framing, gestures, and emotional speech. 📺 Watch the sample: https://drive.google.com/drive/folders/1QtC2il-Qb62nZNJlOtG8WvOf-SSkSGM-?usp=drive_link – Video Preview Dataset Pages: 🐙 GitHub: github.com/MaratDV 📊 DataHub: datahub.io/@MaratDV/storytelling-video-dataset email chinzad@gmail.com or telegram @Marat_DV