Highlight detection has the potential to significantly ease video browsing, but existing methods often suffer from expensive supervision requirements, where human viewers must manually identify highlights in training videos. We propose a scalable unsupervised solution that exploits video duration as an implicit supervision signal. Our key insight is that video segments from shorter user-generated videos are more likely to be highlights than those from longer videos, since users tend to be more selective about the content when capturing shorter videos. Leveraging this insight, we introduce a novel ranking framework that prefers segments from shorter videos, while properly accounting for the inherent noise in the (unlabeled) training data. We use it to train a highlight detector with 10M hashtagged Instagram videos. In experiments on two challenging public video highlight detection benchmarks, our method substantially improves the state-of-the-art for unsupervised highlight detection.
We consider the problem of automatic highlight-detection in video game streams. Currently, the vast majority of highlight-detection systems for games are triggered by the occurrence of hard-coded game events (e.g., score change, end-game), while most advanced tools and techniques are based on detection of highlights via visual analysis of game footage. We argue that in the context of game streaming, events that may constitute highlights are not only dependent on game footage, but also on social signals that are conveyed by the streamer during the play session (e.g., when interacting with viewers, or when commenting and reacting to the game). In this light, we present a multi-view unsupervised deep learning methodology for novelty-based highlight detection. The method jointly analyses both game footage and social signals such as the players facial expressions and speech, and shows promising results for generating highlights on streams of popular games such as Player Unknown's Battlegrounds.
We present an approach for identifying picturesque highlights from large amounts of egocentric video data. Given a set of egocentric videos captured over the course of a vacation, our method analyzes the videos and looks for images that have good picturesque and artistic properties. We introduce novel techniques to automatically determine aesthetic features such as composition, symmetry and color vibrancy in egocentric videos and rank the video frames based on their photographic qualities to generate highlights. Our approach also uses contextual information such as GPS, when available, to assess the relative importance of each geographic location where the vacation videos were shot. Furthermore, we specifically leverage the properties of egocentric videos to improve our highlight detection. We demonstrate results on a new egocentric vacation dataset which includes 26.5 hours of videos taken over a 14 day vacation that spans many famous tourist destinations and also provide results from a user-study to access our results.
We propose a new deep network for audio event recognition, called AENet. In contrast to speech, sounds coming from audio events may be produced by a wide variety of sources. Furthermore, distinguishing them often requires analyzing an extended time period due to the lack of clear sub-word units that are present in speech. In order to incorporate this long-time frequency structure of audio events, we introduce a convolutional neural network (CNN) operating on a large temporal input. In contrast to previous works this allows us to train an audio event detection system end-to-end. The combination of our network architecture and a novel data augmentation outperforms previous methods for audio event detection by 16%. Furthermore, we perform transfer learning and show that our model learnt generic audio features, similar to the way CNNs learn generic features on vision tasks. In video analysis, combining visual features and traditional audio features such as MFCC typically only leads to marginal improvements. Instead, combining visual features with our AENet features, which can be computed efficiently on a GPU, leads to significant performance improvements on action recognition and video highlight detection. In video highlight detection, our audio features improve the performance by more than 8% over visual features alone.
With the prevalence of video sharing, there are increasing demands for automatic video digestion such as highlight detection. Recently, platforms with crowdsourced time-sync video comments have emerged worldwide, providing a good opportunity for highlight detection. However, this task is non-trivial: (1) time-sync comments often lag behind their corresponding shot; (2) time-sync comments are semantically sparse and noisy; (3) to determine which shots are highlights is highly subjective. The present paper aims to tackle these challenges by proposing a framework that (1) uses concept-mapped lexical-chains for lag calibration; (2) models video highlights based on comment intensity and combination of emotion and concept concentration of each shot; (3) summarize each detected highlight using improved SumBasic with emotion and concept mapping. Experiments on large real-world datasets show that our highlight detection method and summarization method both outperform other benchmarks with considerable margins.
We address the problem of highlight detection from a 360 degree video by summarizing it both spatially and temporally. Given a long 360 degree video, we spatially select pleasantly-looking normal field-of-view (NFOV) segments from unlimited field of views (FOV) of the 360 degree video, and temporally summarize it into a concise and informative highlight as a selected subset of subshots. We propose a novel deep ranking model named as Composition View Score (CVS) model, which produces a spherical score map of composition per video segment, and determines which view is suitable for highlight via a sliding window kernel at inference. To evaluate the proposed framework, we perform experiments on the Pano2Vid benchmark dataset and our newly collected 360 degree video highlight dataset from YouTube and Vimeo. Through evaluation using both quantitative summarization metrics and user studies via Amazon Mechanical Turk, we demonstrate that our approach outperforms several state-of-the-art highlight detection methods. We also show that our model is 16 times faster at inference than AutoCam, which is one of the first summarization algorithms of 360 degree videos
Highlight detection models are typically trained to identify cues that make visual content appealing or interesting for the general public, with the objective of reducing a video to such moments. However, the "interestingness" of a video segment or image is subjective. Thus, such highlight models provide results of limited relevance for the individual user. On the other hand, training one model per user is inefficient and requires large amounts of personal information which is typically not available. To overcome these limitations, we present a global ranking model which conditions on each particular user's interests. Rather than training one model per user, our model is personalized via its inputs, which allows it to effectively adapt its predictions, given only a few user-specific examples. To train this model, we create a large-scale dataset of users and the GIFs they created, giving us an accurate indication of their interests. Our experiments show that using the user history substantially improves the prediction accuracy. On our test set of 850 videos, our model improves the recall by 8% with respect to generic highlight detectors. Furthermore, our method proves more precise than the user-agnostic baselines even with just one person-specific example.
롱폼 비디오에 대한 하이라이트 감지는 실용적으로 매우 중요하지만, 적절한 벤치마크가 부족하여 대부분의 기존 방법은 여전히 숏폼 콘텐츠로 제한되어 있습니다. 이러한 격차를 해소하기 위해, 우리는 SVHighlights를 소개합니다. 이는 본 연구진이 아는 한, 여러 스포츠 종목에 걸쳐 각각 1시간 이상의 길이를 가진 초장 스포츠 비디오에서의 하이라이트 감지를 위한 첫 번째 벤치마크입니다. SVHighlights는 데이터셋 생성 파이프라인을 사용하여 전체 스포츠 비디오와 이에 대응하는 공식 하이라이트 비디오 쌍으로 구성되었으며, 이를 통해 기존의 클립 단위 주목도 주석 없이 확장 가능한 레이블 생성이 가능합니다. 이 벤치마크는 평균 길이 2.00시간, 총 640.18시간의 320개 비디오로 구성되어 있어 기존 데이터셋을 크게 초과합니다. 기존 방법들은 긴 비디오에서도 근본적인 과제에 직면합니다. 짧은 클립으로 학습된 모델은 시간 단위의 콘텐츠로 일반화하지 못하며, 클립 수준의 점수 매기기는 하이라이트를 식별하는 데 필요한 더 넓은 문맥이 부족합니다. 이를 해결하고 강력한 기준선(baseline)을 제공하기 위해, 우리는 TF-SELECTOR를 제시합니다. 이는 동일한 의미적 콘텐츠를 공유하는 인접한 샷을 병합하여 각 비디오를 문맥 인식 세그먼트로 나누는 학습 없는 세그먼트 기반 접근법으로, 비주얼 캡션, 대본, 오디오 볼륨을 포함한 멀티모달 입력을 사용하는 대규모 언어 모델(LLM)을 통해 세그먼트 수준의 중요도 점수를 예측합니다. 실험 결과, TF-SELECTOR는 비디오 시간적 그라운딩(Video Temporal Grounding, VTG) 튜닝된 기준선들과 비교하여 대부분의 지표에서 우수한 성능을 달성했으며, HIT@1에서 +2.50, HIT@K에서 +4.04, IoU에서 +2.95의 향상을 보였습니다. 이러한 결과는 SVHighlights가 롱폼 하이라이트 감지를 위한 도전적인 테스트베드임을 확립하고, 간단한 세그먼트 기반 전략이 시간 단위의 비디오로 효과적으로 확장될 수 있음을 보여줍니다.