태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.
A crucial task of Video Understanding is to recognise and localise (in space and time) different actions or events appearing in the video.
출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.
Moviescope is a large-scale dataset of 5,000 movies with corresponding video trailers, posters, plots and metadata. Moviescope is based on the IMDB 5000 dataset consisting of 5.043 movie records. It is augmented by crawling video trailers associated with each movie from YouTube and text plots from Wikipedia. Source: Moviescope: Large-scale Analysis of Movies using Multiple Modalities
Contains ~9K videos of human agents performing various actions, annotated with 3 types of commonsense descriptions.
ChronoMagic with 2265 metamorphic time-lapse videos, each accompanied by a detailed caption.
We introduce InfiniBench a comprehensive benchmark for very long video understanding, which presents 1) The longest video duration, averaging 76.34 minutes; 2) The largest number of question-answer pairs, 108.2K; 3) Diversity in questions that examine nine different skills and include both multiple-choice questions and open-ended questions; 4) Humancentric, as the video sources come from movies and daily TV shows, with specific human-level question designs such as Movie Spoiler Questions that require critical thinking and comprehensive understanding. Using InfiniBench, we comprehensively evaluate existing Large MultiModality Models (LMMs) on each skill, including the commercial model Gemini 1.5 Flash and the open-source models. The evaluation shows significant challenges in our benchmark.Our results show that the best AI models such Gemini struggles to perform well with 42.72% average accuracy and 2.71 out of 5 average score.
LIVE Livestream is a database for Video Quality Assessment (VQA), specifically designed for live streaming VQA research. The dataset is called the Laboratory for Image and Video Engineering (LIVE) Live stream Database. The LIVE Livestream Database includes 315 videos of 45 contents impaired by 6 types of distortions.
The MLB-YouTube dataset is a new, large-scale dataset consisting of 20 baseball games from the 2017 MLB post-season available on YouTube with over 42 hours of video footage. The dataset consists of two components: segmented videos for activity recognition and continuous videos for activity classification. It is quite challenging as it is created from TV broadcast baseball games where multiple different activities share the camera angle. Further, the motion/appearance difference between the various activities is quite small. Source: https://github.com/piergiaj/mlb-youtube Image Source: https://github.com/piergiaj/mlb-youtube
N-Digit MNIST is a multi-digit MNIST-like dataset.
This dataset has the following citation: M. Soliman, M. Kamal, M. Nashed, Y. Mostafa, B. Chawky, D. Khattab, “ Violence Recognition from Videos using Deep Learning Techniques”, Proc. 9th International Conference on Intelligent Computing and Information Systems (ICICIS'19), Cairo, pp. 79-84, 2019. please use it in case of using the dataset in research or engineering purpose ) when we start our Graduation Project Violence Recognition from Videos we found that there is shortage in available datasets related to violence between individuals so we decide to create new big dataset with variety of scenes Content Our Dataset Contains 1000 Violence and 1000 non-violence videos collected from youtube videos, violence videos in our dataset contain many real street fights situations in several environments and conditions. also non-violence videos from our dataset are collected from many different human actions like sports, eating, walking …etc.
Comprises of 171,191 video segments from 346 high-quality soccer games. The database contains 702,096 bounding boxes, 37,709 essential event labels with time boundary and 17,115 highlight annotations for object detection, action recognition, temporal action localization, and highlight detection tasks. Source: SoccerDB: A Large-Scale Database for Comprehensive Video Understanding
Collects dense per-video-shot concept annotations. Source: Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach
A novel dataset that represents complex conversational interactions between two individuals via 3D pose. 8 pairwise interactions describing 7 separate conversation based scenarios were collected using two Kinect depth sensors. Source: From Pose to Activity: Surveying Datasets and Introducing CONVERSE
DeepSportradar is a benchmark suite of computer vision tasks, datasets and benchmarks for automated sport understanding. DeepSportradar currently supports four challenging tasks related to basketball: ball 3D localization, camera calibration, player instance segmentation and player re-identification. For each of the four tasks, a detailed description of the dataset, objective, performance metrics, and the proposed baseline method are provided. Source: DeepSportradar-v1: Computer Vision Dataset for Sports Understanding with High Quality Annotations Image Source: https://github.com/deepsportradar/instance-segmentation-challenge
Largest, first-of-its-kind, in-the-wild, fine-grained workout/exercise posture analysis dataset, covering three different exercises: BackSquat, Barbell Row, and Overhead Press. Seven different types of exercise errors are covered. Unlabeled data is also provided to facilitate self-supervised learning.
Frame-to-frame video alignment/synchronization
A multilingual, multimodal and multi-aspect, expertly-annotated dataset of diverse short videos extracted from short-video social media platform - Moj. 3MASSIV comprises of 50k short videos (~20 seconds average duration) and 100K unlabeled videos in 11 different languages and captures popular short video trends like pranks, fails, romance, comedy expressed via unique audio-visual formats like self-shot videos, reaction videos, lip-synching, self-sung songs, etc.
BibTeX @article{yuan2024chronomagic, title={Chronomagic-bench: A benchmark for metamorphic evaluation of text-to-time-lapse video generation}, author={Yuan, Shenghai and Huang, Jinfa and Xu, Yongqi and Liu, Yaoyang and Zhang, Shaofeng and Shi, Yujun and Zhu, Ruijie and Cheng, Xinhua and Luo, Jiebo and Yuan, Li}, journal={arXiv preprint arXiv:2406.18522}, year={2024} } We provide a database containing shot scale annotations (i.e., the apparent distance of the camera from the subject of a filmed scene) for more than 792,000 image frames. Frames belong to 124 full movies from the entire filmographies by 6 important directors: Martin Scorsese, Jean-Luc Godard, Béla Tarr, Federico Fellini, Michelangelo Antonioni, and Ingmar Bergman. Each frame, extracted from videos at 1 frame per second, is annotated on the following scale categories: Extreme Close Up (ECU), Close Up (CU), Medium Close Up (MCU), Medium Shot (MS), Medium Long Shot (MLS), Long Shot (LS), Extreme Long Shot (ELS), Foreground Shot (FS), and Insert Shots (IS). Two independent coders annotated all frames from the 124 movies, whilst a third one checked their coding and made decisions in cases of disagreement. The CineScale database enables AI-driven interpretation of shot scale data and opens to a large set of research activities related to the automatic visual analysis of cinematic material, such as the automatic recognition of the director’s style, or the unfolding of the relationship between shot scale and the viewers’ emotional experience. To these purposes, we also provide the model and the code for building a Convolutional Neural Network (CNN) architecture for automated shot scale recognition. All this material is provided through the project website, where video frames can also be requested to authors, for research purposes under fair use.
Contains annotations of human activity with different sub-actions, e.g., activity Ping-Pong with four sub-actions which are pickup-ball, hit, bounce-ball and serve. Source: Egok360: A 360 Egocentric Kinetic Human Activity Video Dataset
Provide: * 1,105 lifestyle videos that span diverse scenarios. * Each video in the MECD dataset contains 4 to 11 events, with a minimum of 2 premise events exhibiting causal relations with the last one. * The “duration”, “sentence”, and “timestamps” attributes of each video. * A new attribute “relation” to evaluate the video causal discovery ability of the model, and support the supervised training.
Stanford-ECM is an egocentric multimodal dataset which comprises about 27 hours of egocentric video augmented with heart rate and acceleration data. The lengths of the individual videos cover a diverse range from 3 minutes to about 51 minutes in length. A mobile phone was used to collect egocentric video at 720x1280 resolution and 30 fps, as well as triaxial acceleration at 30Hz. The mobile phone was equipped with a wide-angle lens, so that the horizontal field of view was enlarged from 45 degrees to about 64 degrees. A wrist-worn heart rate sensor was used to capture the heart rate every 5 seconds. The phone and heart rate monitor was time-synchronized through Bluetooth, and all data was stored in the phone’s storage. Piecewise cubic polynomial interpolation was used to fill in any gaps in heart rate data. Finally, data was aligned to the millisecond level at 30 Hz. Source: http://ai.stanford.edu/~syyeung/ecm_dataset/egocentric_multimodal.html Image Source: http://ai.stanford.edu/~syyeung/ecm_dataset/egocentric_multimodal.html
git clone https://huggingface.co/yzy666/SVBench Intended Use - Evaluate long-context reasoning capabilities of Large Vision-Language Models (LVLMs) in streaming video scenarios. - Benchmark temporal understanding through multi-turn dialogues. - Support research in streaming video QA, activity forecasting, and interactive AI assistants. Direct Use - Training or fine-tuning LVLMs for streaming video tasks. - Testing model robustness in handling dynamic, time-sensitive queries. - Comparative analysis of open-source vs. closed-source LVLMs. Restrictions - Videos and annotations are for research purposes only. - Users must comply with the original licenses of datasets. ## Dataset Structure - Folder Structure Tree: SVBench/ ├── Con/ │ ├── Con_EN/ │ └── Con_ZH/ ├── Dialogue/ │ ├── Dialogue_EN/ │ └── Dialogue_ZH/ ├── Meta/ │ ├── Meta_EN/ │ └── Meta_ZH/ ├── Path/ ├── Src/ ├── Streaming/ │ ├── Streaming_EN/ │ └── Streaming_ZH/ ├── Video/ └── Your_Model_Name/ ├── dialogue/ │ └── --NDulaHyrE.json └── streaming/ └── -4h8cuweoKo.json - Dataset Division: - - Training Set: 42,605 QA pairs and 1,153 videos - Testing Set: 7,374 QA pairs and 200 videos ## Data Fields | Key | Description | | ------------------------- | ------------------------------------------------------------ | | Video_Name | Unique identifier or title of the video file. | | Sort_of_Set | Category or type of the dataset subset (e.g., "Train", "Test"). | | Path_of_QandA | File path to the question-answer pairs. | | Path_of_Con | Path to relationship files. | | Path_of_StreamingPathData | Path to Q&A sequence for streaming evaluation. Each streaming path contains all Q&A sequences in streaming order within the path. | | Path_of_Dialogue | Path to Q&A sequence for dialogue evaluation. Each dialogue contains all Q&A sequences in order within the dialogue. | | Path_of_Streaming | Path to Q&A sequence for streaming evaluation represented only by serial numbers (e.g., the path [[0,0],[1,2],[2,3]...] indicates starting from the 1st question of the 1st chain, then proceeding to the 3rd question of the 2nd chain, and then to the 4th question of the 3rd chain, and so on). | | Path_of_Video | Absolute file path to the raw video file. | | Video_Duration | Total duration of the video in seconds. | | Source_of_Dataset | Origin of the dataset. | ## Leaderboard Submission Submit results via https://forms.gle/Rmi6u4WGhyEZ2X7g8. Submission Instructions: 1. Save result files following the case structure under the SVBench/[Your_Model_Name] directory 2. Compress the entire [Your_Model_Name] folder into a ZIP archive 3. Upload the generated ZIP file through this submission portal Important Notes: - Maintain the exact directory hierarchy: [Your_Model_Name]/[dialogue/streaming]/[Video_Name.json] - Ensure your model name follows exact capitalization (case-sensitive) - Package only the required result files (avoid including source code/executables) Submission Form img ## Dataset Annotation See GitHub repository for details. Semi-automated annotation using a hybrid approach: - Automatic QA Generation: Leveraging GPT-4 to generate dialogue chains based on video transcripts. - Human Verification: Annotators validate temporal links and refine QA pairs for consistency. - Temporal Linking: Artificial alignment of QA chains across video segments. - Quality Control: Inter-annotator agreement >90% for temporal links. ## Citation If you find our data useful, please consider citing our work! @article{yang2025svbench, title={SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding}, author={Yang, Zhenyu and Hu, Yuhang and Du, Zemin and Xue, Dizhan and Qian, Shengsheng and Wu, Jiahong and Yang, Fan and Dong, Weiming and Xu, Changsheng}, journal={arXiv preprint arXiv:2502.10810}, year={2025} }Vision-based Fallen Person (VFP290K) is a novel, large-scale dataset for the detection of fallen persons composed of fallen person images collected in various real-world scenarios. VFP290K consists of 294,714 frames of fallen persons extracted from 178 videos, including 131 scenes in 49 locations.
VTC is a large-scale multimodal dataset containing video-caption pairs (~300k) alongside comments that can be used for multimodal representation learning.
This large collection of over 161,000 video-label pairs of video clips, shows humans drawing letters and digits in the air, and is used to evaluate a model’s ability to classify articulated motions correctly. Unlike existing video datasets, AirLetters’ accurate classification predictions rely on discerning motion patterns and integrating information presented by the video over time (i.e., over many frames of video). That study revealed that while trivial for humans, accurate representations of complex articulated motions remain an open problem for end-to-end learning for video understanding models.
The feature files are named with the youtube IDs. https://drive.google.com/drive/folders/10-6hkQxMKMGwLXANxfPRE7xw5PKiMjLn?usp=sharing
CausalChaos! is a dataset for causal video question answering. It is based on Tom and Jerry cartoons. It features longer causal chains embedded in dynamic visual scenes. It also features challenging incorrect options, especially, Causal Confusion set which contains causally confounding incorrect options. All these factors prove to be challenging for current VLMs and other traditional Video Question Answering models.
BibTeX @article{yuan2024chronomagic, title={Chronomagic-bench: A benchmark for metamorphic evaluation of text-to-time-lapse video generation}, author={Yuan, Shenghai and Huang, Jinfa and Xu, Yongqi and Liu, Yaoyang and Zhang, Shaofeng and Shi, Yujun and Zhu, Ruijie and Cheng, Xinhua and Luo, Jiebo and Yuan, Li}, journal={arXiv preprint arXiv:2406.18522}, year={2024} } CinePile is a question-answering-based, long-form video understanding dataset. It has been created using advanced large language models (LLMs) with human-in-the-loop pipeline leveraging existing human-generated raw data. It consists of approximately 300,000 training data points and 5,000 test data points.
This spatio-temporal actions dataset for video understanding consists of 4 parts: original videos, cropped videos, video frames, and annotation files. This dataset uses a proposed new multi-person annotation method of spatio-temporal actions. First, we use ffmpeg to crop the videos and frame the videos; then use yolov5 to detect human in the video frame, and then use deep sort to detect the ID of the human in the video frame. By processing the detection results of yolov5 and deep sort, we can get the annotation file of the spatio-temporal action dataset to complete the work of customizing the spatio-temporal action dataset.
The i3-video dataset contains "is-it-instructional" annotations for 6.4k videos from Youtube-8M. The videos are considered to be instructional if they focus on real-world human actions accompanied by procedural language that explains what’s happening on screen in reasonable details. Source: i3-video
The IAW dataset contains 420 Ikea furniture pieces from 14 common categories e.g. sofa, bed, wardrobe, table, etc. Each piece of furniture comes with one or more user instruction manuals, which are first divided into pages and then further divided into independent steps cropped from each page (some pages contain more than one step and some pages do not contain instructions). There are 8568 pages and 8263 steps overall, on average 20.4 pages and 19.7 steps for each piece of furniture. We crawled YouTube to find videos corresponding to these instruction manuals and as such the conditions in the videos are diverse on many aspects e.g. duration, resolution, first- or third-person view, camera pose, background environment, number of assemblers, etc. The IAW dataset contains 1005 raw videos with a length of around 183 hours in total. Among them, approximately 114 hours of content are labeled as 15649 actions to match the corresponding step in the corresponding manual.
Kinetics-GEB+ (Generic Event Boundary Captioning, Grounding and Retrieval) is a dataset that consists of over 170k boundaries associated with captions describing status changes in the generic events in 12K videos.
A benchmark that focuses on the sampling dilemma in long-video tasks. The LSDBench dataset is designed to evaluate the sampling efficiency of long-video VLMs. It consists of multiple-choice question-answer pairs based on hour-long videos, focusing on dense and short-duration actions with high Necessary Sampling Density (NSD).
Music recommendation for videos attracts growing interest in multi-modal research. However, existing systems focus primarily on content compatibility, often ignoring the users’ preferences. Their inability to interact with users for further refinements or to provide explanations leads to a less satisfying experience. We address these issues with MuseChat, a first-of-its-kind dialogue-based recommendation system that personalizes music suggestions for videos. Our system consists of two key functionalities with associated modules: recommendation and reasoning. The recommendation module takes a video along with optional information including previous suggested music and user’s preference as inputs and retrieves an appropriate music matching the context. The reasoning module, equipped with the power of Large Language Model (Vicuna-7B) and extended to multi-modal inputs, is able to provide reasonable explanation for the recommended music. To evaluate the effectiveness of MuseChat, we build a large-scale dataset, conversational music recommendation for videos, that simulates a two-turn interaction between a user and a recommender based on accurate music track information. Experiment results show that MuseChat achieves significant improvements over existing video-based music retrieval methods as well as offers strong interpretability and interactability. The dataset of this work is available at https://dongzhikang.github.io/musechat.
Placepedia contains 240K places with 35M images from all over the world. Each place is associated with its district, city/town/village, state/province, country, continent, and a large amount of diverse photos. Both administrative areas and places have rich side information, e.g. discription, population, category, function. In addition, two cleaned subsets (Places-Coarse and Places-Fine) for experiments are provided.
출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.