Papers with Code 아카이브

Video

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
34
평가표
0
보존된 평가 기록
0

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

i3-video

The i3-video dataset contains "is-it-instructional" annotations for 6.4k videos from Youtube-8M. The videos are considered to be instructional if they focus on real-world human actions accompanied by procedural language that explains what’s happening on screen in reasonable details. Source: i3-video

IAW Dataset

The IAW dataset contains 420 Ikea furniture pieces from 14 common categories e.g. sofa, bed, wardrobe, table, etc. Each piece of furniture comes with one or more user instruction manuals, which are first divided into pages and then further divided into independent steps cropped from each page (some pages contain more than one step and some pages do not contain instructions). There are 8568 pages and 8263 steps overall, on average 20.4 pages and 19.7 steps for each piece of furniture. We crawled YouTube to find videos corresponding to these instruction manuals and as such the conditions in the videos are diverse on many aspects e.g. duration, resolution, first- or third-person view, camera pose, background environment, number of assemblers, etc. The IAW dataset contains 1005 raw videos with a length of around 183 hours in total. Among them, approximately 114 hours of content are labeled as 15649 actions to match the corresponding step in the corresponding manual.

Imbalanced-MiniKinetics200

Imbalanced-MiniKinetics200 was proposed by "Minority-Oriented Vicinity Expansion with Attentive Aggregation for Video Long-Tailed Recognition" to evaluate varying scenarios of video long-tailed recognition. Similar to CIFAR-10/100-LT, it utilizes an imbalance factor to construct long-tailed variants of the MiniKinetics200 dataset. Imbalanced-MiniKinetics200 is a subset of Mini-Kinetics-200 consisting of 200 categories which is also a subset of Kinetics400. Both the raw frames and extracted features with ResNet50/101 are provided.

Kinetics-GEB+

Kinetics-GEB+ (Generic Event Boundary Captioning, Grounding and Retrieval) is a dataset that consists of over 170k boundaries associated with captions describing status changes in the generic events in 12K videos.

LAVIB

LAVIB comprises a large collection of high-resolution videos sourced from the web. Metrics are computed for each video's motion magnitudes, luminance conditions, frame sharpness, and contrast. In total, LAVIB includes 283K clips from 17K ultra-HD videos, covering 77.6 hours. Benchmark train, val, and test sets maintain similar video metric distributions. Further splits are also created for out-of-distribution (OOD) challenges, with train and test splits including videos of dissimilar attributes.

LSDBench

A benchmark that focuses on the sampling dilemma in long-video tasks. The LSDBench dataset is designed to evaluate the sampling efficiency of long-video VLMs. It consists of multiple-choice question-answer pairs based on hour-long videos, focusing on dense and short-duration actions with high Necessary Sampling Density (NSD).

Metaphorics

Metaphorics is a newly introduced non-contextual skeleton action dataset. All the datasets introduced so far in the skeleton human action recognition have categories based only on verb-based actions. Source: Metaphorics

MLB

This dataset provides high-resolution videos recorded from three perspectives with more than 110 hours of total playtime showing mice solving complex tasks. We provide frame-level action labels that reflect a mouse's actions (in proximity to, touch, bite, lock, unlock, touch reward) with lockbox mechanisms (lever, stick, ball, sliding door) for 13% of the data.

MODIS AOD (imputed)

Structured atmospheric data for AI/ML Long-term, pre-processed, atmospheric datasets for use in Machine Learning/AI based forecasting. Initially intended to predict AOD, however can be adapted for prediction of other atmospheric particles. Machine Learning ready NumPy* dataset constructed by pre-processing selected atmospheric variables at 5 pressure levels form ERA5 reanalysis (resulting in 35 features) and AOD data from MODIS on board of Aqua and Terra satellites. This is a long-term daily dataset which spans 20 years from 1st Jan 2003 to 31st Dec 2022 and is homogeneously structured into 1ºx1º grid cells. Missing days and AOD values from MODIS were imputed using Lattice Kriging method (Python code used for imputation included as Jupyter Notebook 'Combine_impute_AOD.ipynb'), but raw (unimputed) MODIS data are also available. All datasets were created for a purpose of training Convolutional Neural Network model designed to forecast Saharan dust (DustNet). These datasets can also be used to train other ML models, or indeed to forecast other variables. This dataset was used to train the DustNet model and predict 24-hr ahead AOD. Please see doi: 10.5281/zenodo.10722953 for further details on predicting AOD and the DustNet model code. *datasets are NumPy arrays (v1.23) created in Python v3.8.18.

Moving Symbols

A parameterized synthetic dataset called Moving Symbols to support the objective study of video prediction networks. Source: A Dataset To Evaluate The Representations Learned By Video Prediction Models

Mr. HiSum

Mr. HiSum is a large-scale video highlight detection and summarization dataset, which contains 31,892 videos selected from YouTube-8M dataset and reliable frame importance score labels aggregated from 50,000+ users per video.

MSVD-Indonesian

MSVD-Indonesian is derived from the MSVD dataset, which is obtained with the help of a machine translation service. This dataset can be used for multimodal video-text tasks, including text-to-video retrieval, video-to-text retrieval, and video captioning. Same as the original English dataset, the MSVD-Indonesian dataset contains about 80k video-text pairs.

Multi-Ego

A new multi-view egocentric dataset, Multi-Ego. The dataset is recorded simultaneously by three cameras, covering a wide variety of real-life scenarios. The footage is annotated by multiple individuals under various summarization configurations, with a consensus analysis ensuring a reliable ground truth. Source: Multi-Stream Dynamic Video Summarization

Multimodal PISA

Dataset for multimodal skills assessment focusing on assessing piano player’s skill level. Annotations include player's skills level, and song difficulty level. Bounding box annotations around pianists' hands are also provided.

MultiSenseBadminton

The sports industry is witnessing an increasing trend of utilizing multiple synchronized sensors for player data collection, enabling personalized training systems with multi-perspective real-time feedback. Badminton could benefit from these various sensors, but there is a scarcity of comprehensive badminton action datasets for analysis and training feedback. Addressing this gap, this paper introduces a multi-sensor badminton dataset for forehand clear and backhand drive strokes, based on interviews with coaches for optimal usability. The dataset covers various skill levels, including beginners, intermediates, and experts, providing resources for understanding biomechanics across skill levels. It encompasses 7,763 badminton swing data from 25 players, featuring sensor data on eye tracking, body tracking, muscle signals, and foot pressure. The dataset also includes video recordings, detailed annotations on stroke type, skill level, sound, ball landing, and hitting location, as well as survey and interview data. We validated our dataset by applying a proof-of-concept machine learning model to all annotation data, demonstrating its comprehensive applicability in advanced badminton training and research.

MultiSum

MultiSum is a dataset for multimodal summarization (MSMO). It consists of 17 categories and 170 subcategories to encapsulate a diverse array of real-world scenarios. The dataset features: 1)Human-validated summaries for both video and textual content, providing superior human instruction and labels for multimodal learning. 2) Comprehensively and meticulously arranged categorization, spanning 17 principal categories and 170 subcategories to encapsulate a diverse array of real-world scenarios. 3) Benchmark tests performed on the proposed dataset to assess varied tasks and methods, including video temporal segmentation, video summarization, text summarization, and multimodal summarization. Source: MultiSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos Image Source: MultiSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos

MuseChat Dataset

Music recommendation for videos attracts growing interest in multi-modal research. However, existing systems focus primarily on content compatibility, often ignoring the users’ preferences. Their inability to interact with users for further refinements or to provide explanations leads to a less satisfying experience. We address these issues with MuseChat, a first-of-its-kind dialogue-based recommendation system that personalizes music suggestions for videos. Our system consists of two key functionalities with associated modules: recommendation and reasoning. The recommendation module takes a video along with optional information including previous suggested music and user’s preference as inputs and retrieves an appropriate music matching the context. The reasoning module, equipped with the power of Large Language Model (Vicuna-7B) and extended to multi-modal inputs, is able to provide reasonable explanation for the recommended music. To evaluate the effectiveness of MuseChat, we build a large-scale dataset, conversational music recommendation for videos, that simulates a two-turn interaction between a user and a recommender based on accurate music track information. Experiment results show that MuseChat achieves significant improvements over existing video-based music retrieval methods as well as offers strong interpretability and interactability. The dataset of this work is available at https://dongzhikang.github.io/musechat.

MVS1K

Contains about 1, 000 videos from 10 queries and their video tags, manual annotations, and associated web images. Source: Query-Aware Sparse Coding for Multi-Video Summarization

Omni-MOT

The Omni-MOT is realistic CARLA based large-scale dataset with over 14M frames for multiple vehicle tracking . The dataset comprises 14M+ frames, 250K tracks, 110 million bounding boxes, three weather conditions, three crowd levels and three camera views in five simulated towns. Source: Simultaneous Detection and Tracking with Motion Modelling for Multiple Object Tracking

Open-HypermotionX

🏃‍♂️ Open-HypermotionX Dataset Open-Hypermotion is a large-scale, high-quality dataset designed for training and evaluating pose-guided human image animation models, with a special focus on complex, dynamic human motions (Hypermotion), such as flips, spins, and acrobatics. ### 🔍 Key Features - Multimodal Annotations: - 2D human pose sequences - Reference images for appearance control - Video clips of target motions - Text descriptions of actions - Motion Diversity: - Covers both common activities and high-difficulty dynamic actions - Task-Specific Subsets: - Pose-guided animation

OpenS2V-5M

We create the first open-source large-scale S2V generation dataset OpenS2V-5M, which consists of five million high-quality 720P subject-text-video triples. To ensure subject-information diversity in our dataset by, we (1) segmenting subjects and building pairing information via cross-video associations and (2) prompting GPT-Image on raw frames to synthesize multi-view representations. The dataset supports both Subject-to-Video and Text-to-Video generation tasks.

Placepedia

Placepedia contains 240K places with 35M images from all over the world. Each place is associated with its district, city/town/village, state/province, country, continent, and a large amount of diverse photos. Both administrative areas and places have rich side information, e.g. discription, population, category, function. In addition, two cleaned subsets (Places-Coarse and Places-Fine) for experiments are provided.

SAT-MTB-VSR

SAT-MTB-VSR is a large-scale dataset for satellite video super-resolution made from original videos of Jilin-1, which is a subset of the satellite video multitasking dataset SAT-MTB. The dataset is cropped from 18 videos captured by the Jilin-1 video satellite, covering a wide range of terrains, such as cities, docks, airports, suburbs, forests, and deserts, with a resolution of about 1 m. And the videos contain dynamic scenes, such as moving cars, airplanes, trains, and ships, which test the ability of the VSR method to deal with moving targets of different sizes and speeds. At the same time, due to the motion of the satellite, the video contains changes in viewing angle and lighting. It contains 431 videos, each of which is 100 consecutive frames, of which 413 are used as the training set and 18 as the validation set, and all the 18 validation sets are from different original videos, while the size of the images is 640 × 640. These images are downsampled 4× by bicubic interpolation to get 160 × 160 low-resolution images, thus obtaining the LR-HR training pairs.

SOMPT22

SOMPT22 is a multi-object tracking (MOT) benchmark focused on surveillance-style pedestrian tracking. * 22 long video sequences (static pole-mounted cameras, 6 – 8 m height) * ~51 k annotated frames with bounding boxes + unique track IDs * Outdoor scenes with illumination changes, partial occlusions and appearance similarity * Single class: person * Split files ready for training/validation and standard MOT evaluation tools SOMPT22 aims to complement generic MOTChallenge-style datasets by stressing long-term ID maintenance under sparse-to-medium crowd density instead of dense, short clips. Homepage → https://sompt22.github.io Download → Google Drive link in the homepage Citation → ```bibtex @misc{simsek2022sompt22, author = {Simsek, Fatih Emre and Cigla, Cevahir and Kayabol, Koray}, title = {SOMPT22: A Surveillance Oriented Multi-Pedestrian Tracking Dataset}, year = {2022}, eprint = {2208.02580}, archivePrefix = {arXiv}, primaryClass = {cs.CV} }

SOTVerse

SOTVerse is a user-defined task space of single object tracking. It allows users to customize SOT tasks according to their research purposes, which on the one hand makes research more targeted, and on the other hand can significantly improve the efficiency of research.

SRI-APPROVE Fine-Grained Video Classification

APPROVE consists of curated YouTube videos annotated with educational content. APPROVE consists of 193 hours of expert-annotated videos with 19 classes (7 literacy codes, 11 math, and background) and each video is associated with approximately 3 labels on average.

STVD-PVCD

STVD is the largest public dataset on the PVCD task. It was constituted with about 83 thousands of videos having in total of more than 10 thousands of hours duration and including more than 420 thousands of video copy pairs. It offers different test sets for a fine performance characterization (frame degradation, global transformation, video speeding, etc.) with a frame level annotation for the real-time detection and video alignment. Baseline comparisons were reported to show a room for improvement. More information about the STVD dataset can be found into the publications [1, 2]. [1] V.H. Le, M. Delalandre and D. Conte. A large-Scale TV Dataset for partial video copy detection. International Conference on Image Analysis and Processing (ICIAP), Lecture Notes in Computer Science (LNCS), vol 13233, pp. 388-399, 2022. http://mathieu.delalandre.free.fr/publications/ICIAP2022.pdf [2] V.H. Le, M. Delalandre and D. Conte. Une large base de données pour la détection de segments de vidéos TV. Journées Francophones des Jeunes Chercheurs en Vision par Ordinateur (ORASIS), 2021. http://mathieu.delalandre.free.fr/publications/ORASIS2021.pdf

SynoClip

SynoClip Dataset The SynoClip dataset is a comprehensive and standard dataset specifically designed for the video synopsis task. It consists of six videos, ranging from 8 to 45 minutes, captured from outdoor-mounted surveillance cameras. This dataset is annotated with tracking information, making it an ideal resource not only for video synopsis but also for related tasks such as object detection in videos and multi-object tracking. ### Key Features: - Outdoor Surveillance Footage: Captured from outdoor-mounted cameras, the dataset reflects real-world surveillance conditions. - Tracking Annotations: Each video comes with detailed tracking annotations, enabling a fair comparison of video synopsis models' performance. - Multi-Purpose Utility: While primarily intended for video synopsis, the dataset can also be used for training and evaluating object detection and tracking models in videos. ### Download the Dataset You can download the SynoClip dataset from the following Google Drive link: Download SynoClip Dataset

TLFM dataset

TLFM dataset structured in sequences of at least nine timesteps. The dataset includes 9696 images of both brightfield and green fluorescent protein channels at a resolution of 256 × 256. Dataset for multi-domain (BF and GFP) microscopy image sequence generation.

Trailers12k

Trailers12k is a movie trailer dataset comprised of 12,000 titles associated to ten genres. It distinguishes from other datasets by its collection procedure aimed at providing a high-quality publicly available dataset.

VADD

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

VCG+112K

Video-ChatGPT introduces the VideoInstruct100K dataset, which employs a semi-automatic annotation pipeline to generate 75K instruction-tuning QA pairs. To address the limitations of this annotation process, we present \ourdata~dataset developed through an improved annotation pipeline. Our approach improves the accuracy and quality of instruction tuning pairs by improving keyframe extraction, leveraging SoTA large multimodal models (LMMs) for detailed descriptions, and refining the instruction generation strategy.

ViDAS

100 videos with varying danger levels (on a scale of 0-10) and different scenarios, annotated by 18 human annotators using our annotation pipeline to represent human perception and respective Vision Language model summaries for each of the videos as benchmarks for testing LLMs' danger perceptions.

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.