Video Summarization

The SumMe dataset is a video summarization dataset consisting of 25 videos, each annotated with at least 15 human summaries (390 in total). Source: https://gyglim.github.io/me/vsum/index.html Image Source: https://gyglim.github.io/me/vsum/index.html

Introduced by Song et al. in TVSum: Summarizing web videos using titles. The TVSum dataset comprises 50 videos, with durations ranging from 1 to 11 minutes. These videos belong to 10 different categories associated with the TRECVid MED task, with 5 videos in each category, and were collected from YouTube. The video categories include various activities like changing a vehicle tire, making a sandwich, and flash mob gatherings. For annotation, each video was reviewed and rated by 20 users, who assigned frame-level importance scores on a scale from 1 (not important) to 5 (very important).

HyperKvasir dataset contains 110,079 images and 374 videos where it captures anatomical landmarks and pathological and normal findings. A total of around 1 million images and video frames altogether.

The Video2GIF dataset contains over 100,000 pairs of GIFs and their source videos. The GIFs were collected from two popular GIF websites (makeagif.com, gifsoup.com) and the corresponding source videos were collected from YouTube in Summer 2015. IDs and URLs of the GIFs and the videos are provided, along with temporal alignment of GIF segments to their source videos. The dataset shall be used to evaluate GIF creation and video highlight techniques. In addition to the 100K GIF-video pairs, the dataset contains 357 pairs of GIFs and their source videos as the test set. The 357 videos come with a Creative Commons CC-BY license, which allowed the authors to redistribute the material with appropriate credit to make the results on test set reproducible even when some of the videos become unavailable. Source: Video2GIF

VideoXum is an enriched large-scale dataset for cross-modal video summarization. The dataset is built on ActivityNet Captions. The datasets includes three subtasks: Video-to-Video Summarization (V2V-SUM), Video-to-Text Summarization (V2T-SUM), and Video-to-Video&Text Summarization (V2VT-SUM).

Collects dense per-video-shot concept annotations. Source: Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach

Contains 140 videos with multiple human created summaries, which were acquired in a controlled experiment. Source: Diversity-aware Multi-Video Summarization

Ego4D-HCap is a hierarchical video captioning dataset comprised of a three-tier hierarchy of captions: short clip-level captions, medium-length video segment descriptions, and long-range video-level summaries. To construct Ego4D-HCap, we leverage Ego4D, the largest publicly available egocentric video dataset. While Ego4D comes with time-stamped atomic captions and video-segment descriptions spanning up to 5 minutes, it lacks video-level summaries for longer video durations. To address this issue, we annotate a subset of 8,267 Ego4D videos with long-range video summaries, each spanning up to two hours. This enhancement provides a three-level hierarchy of captions.

A short clip of video may contain progression of multiple events and an interesting story line. A human needs to capture both the event in every shot and associate them together to understand the story behind it. In this work, we present a new multi-shot video understanding benchmark Shot2Story with detailed shot-level captions and comprehensive video summaries. To facilitate better semantic understanding of videos, we provide captions for both visual signals and human narrations. We design several distinct tasks including single-shot video and narration captioning, multi-shot video summarization, and video retrieval with shot descriptions. Preliminary experiments show some challenges to generate a long and comprehensive video summary. Nevertheless, the generated imperfect summaries can already significantly boost the performance of existing video understanding tasks such as video question-answering, promoting an underexplored setting of video understanding with detailed summaries.

The dataset is useful for query-adaptive video summarization and annotated with diversity and query-specific relevance labels. Source: Query-adaptive Video Summarization via Quality-aware Relevance Estimation

Mr. HiSum is a large-scale video highlight detection and summarization dataset, which contains 31,892 videos selected from YouTube-8M dataset and reliable frame importance score labels aggregated from 50,000+ users per video.

A new multi-view egocentric dataset, Multi-Ego. The dataset is recorded simultaneously by three cameras, covering a wide variety of real-life scenarios. The footage is annotated by multiple individuals under various summarization configurations, with a consensus analysis ensuring a reliable ground truth. Source: Multi-Stream Dynamic Video Summarization

MultiSum is a dataset for multimodal summarization (MSMO). It consists of 17 categories and 170 subcategories to encapsulate a diverse array of real-world scenarios. The dataset features: 1)Human-validated summaries for both video and textual content, providing superior human instruction and labels for multimodal learning. 2) Comprehensively and meticulously arranged categorization, spanning 17 principal categories and 170 subcategories to encapsulate a diverse array of real-world scenarios. 3) Benchmark tests performed on the proposed dataset to assess varied tasks and methods, including video temporal segmentation, video summarization, text summarization, and multimodal summarization. Source: MultiSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos Image Source: MultiSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos

Contains about 1, 000 videos from 10 queries and their video tags, manual annotations, and associated web images. Source: Query-Aware Sparse Coding for Multi-Video Summarization

This dataset consists of 18 movies with duration range between 10 and 104 minutes leveraged from the OVSD dataset (Rotman et al., 2016). For these videos, the summary length limit is set to be the minimum between 4 minutes and 10% of the video length. Source: ILS-SUMM: ITERATED LOCAL SEARCH FOR UNSUPERVISED VIDEO SUMMARIZATION

SoccerNet-Echoes: A Soccer Game Audio Commentary Dataset.