2D Human Pose Estimation

The DukeMTMC-reID (Duke Multi-Tracking Multi-Camera ReIDentification) dataset is a subset of the DukeMTMC for image-based person re-ID. The dataset is created from high-resolution videos from 8 different cameras. It is one of the largest pedestrian image datasets wherein images are cropped by hand-drawn bounding boxes. The dataset consists 16,522 training images of 702 identities, 2,228 query images of the other 702 identities and 17,661 gallery images. NOTE: This dataset has been retracted. Source: Deep Co-attention based Comparators For Relative Representation Learning in Person Re-identification

JHMDB is an action recognition dataset that consists of 960 video sequences belonging to 21 actions. It is a subset of the larger HMDB51 dataset collected from digitized movies and YouTube videos. The dataset contains video and annotation for puppet flow per frame (approximated optimal flow on the person), puppet mask per frame, joint positions per frame, action label per clip and meta label per clip (camera motion, visible body parts, camera viewpoint, number of people, video quality). Source: Unsupervised Deep Metric Learning via Orthogonality based Probabilistic Loss Image Source: https://arxiv.org/pdf/1712.06316.pdf

The Vimeo-90K is a large-scale high-quality video dataset for lower-level video processing. It proposes three different video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution. Source: https://arxiv.org/pdf/1711.09078.pdf Image Source: http://toflow.csail.mit.edu/

MPI (Max Planck Institute) Sintel is a dataset for optical flow evaluation that has 1064 synthesized stereo images and ground truth data for disparity. Sintel is derived from open-source 3D animated short film Sintel. The dataset has 23 different scenes. The stereo images are RGB while the disparity is grayscale. Both have resolution of 1024×436 pixels and 8-bit per channel. Source: Fast Disparity Estimation using Dense Networks*

The PASCAL Visual Object Classes (VOC) 2012 dataset contains 20 object categories including vehicles, household, animals, and other: aeroplane, bicycle, boat, bus, car, motorbike, train, bottle, chair, dining table, potted plant, sofa, TV/monitor, bird, cat, cow, dog, horse, sheep, and person. Each image in this dataset has pixel-level segmentation annotations, bounding box annotations, and object class annotations. This dataset has been widely used as a benchmark for object detection, semantic segmentation, and classification tasks. The PASCAL VOC dataset is split into three subsets: 1,464 images for training, 1,449 images for validation and a private testing set. Source: Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey Image Source: http://host.robots.ox.ac.uk/pascal/VOC/voc2012/examples/images/sheep_06.jpg

This paper introduces the pipeline to scale the largest dataset in egocentric vision EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version (EPIC-KITCHENS-55), EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments). This collection also enables evaluating the "test of time" - i.e. whether models trained on data collected in 2018 can generalise to new footage collected under the same hypotheses albeit "two years on". The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics.

Grammarly’s Yahoo Answers Formality Corpus (GYAFC) is the largest dataset for any style containing a total of 110K informal / formal sentence pairs. Yahoo Answers is a question answering forum, contains a large number of informal sentences and allows redistribution of data. The authors used the Yahoo Answers L6 corpus to create the GYAFC dataset of informal and formal sentence pairs. In order to ensure a uniform distribution of data, they removed sentences that are questions, contain URLs, and are shorter than 5 words or longer than 25. After these preprocessing steps, 40 million sentences remain. The Yahoo Answers corpus consists of several different domains like Business, Entertainment & Music, Travel, Food, etc. Pavlick and Tetreault formality classifier (PT16) shows that the formality level varies significantly across different genres. In order to control for this variation, the authors work with two specific domains that contain the most informal sentences and show results on training and testing within those categories. The authors use the formality classifier from PT16 to identify informal sentences and train this classifier on the Answers genre of the PT16 corpus which consists of nearly 5,000 randomly selected sentences from Yahoo Answers manually annotated on a scale of -3 (very informal) to 3 (very formal). They find that the domains of Entertainment & Music and Family & Relationships contain the most informal sentences and create the GYAFC dataset using these domains. Source: Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer

Extended GTEA Gaze+ EGTEA Gaze+ is a large-scale dataset for FPV actions and gaze. It subsumes GTEA Gaze+ and comes with HD videos (1280x960), audios, gaze tracking data, frame-level action annotations, and pixel-level hand masks at sampled frames. Specifically, EGTEA Gaze+ contains 28 hours (de-identified) of cooking activities from 86 unique sessions of 32 subjects. These videos come with audios and gaze tracking (30Hz). We have further provided human annotations of actions (human-object interactions) and hand masks. The action annotations include 10325 instances of fine-grained actions, such as "Cut bell pepper" or "Pour condiment (from) condiment container into salad". The hand annotations consist of 15,176 hand masks from 13,847 frames from the videos. Source: http://cbs.ic.gatech.edu/fpv/ Image Source: http://cbs.ic.gatech.edu/fpv/

WikiArt contains painting from 195 different artists. The dataset has 42129 images for training and 10628 images for testing. Source: Adding New Tasks to a Single Network with Weight Transformations using Binary Masks Image Source: https://towardsdatascience.com/the-non-treachery-of-dataset-df1f6cbe577e

MPIIGaze is a dataset for appearance-based gaze estimation in the wild. It contains 213,659 images collected from 15 participants during natural everyday laptop use over more than three months. It has a large variability in appearance and illumination. Source: https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/research/gaze-based-human-computer-interaction/appearance-based-gaze-estimation-in-the-wild Image Source: https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/research/gaze-based-human-computer-interaction/appearance-based-gaze-estimation-in-the-wild

AGORA is a synthetic human dataset with high realism and accurate ground truth. It consists of around 14K training and 3K test images by rendering between 5 and 15 people per image using either image-based lighting or rendered 3D environments, taking care to make the images physically plausible and photoreal. In total, AGORA contains 173K individual person crops. AGORA provides (1) SMPL/SMPL-X parameters and (2) segmentation masks for each subject in images.

This dataset focuses on heavily occluded human with comprehensive annotations including bounding-box, humans pose and instance mask. This dataset contains 13,360 elaborately annotated human instances within 5081 images. With average 0.573 MaxIoU of each person, OCHuman is the most complex and challenging dataset related to human. Source: https://github.com/liruilong940607/OCHumanApi Image Source: https://github.com/liruilong940607/OCHumanApi

VOCASET is a 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio. The dataset has 12 subjects and 480 sequences of about 3-4 seconds each with sentences chosen from an array of standard protocols that maximize phonetic diversity. Source: timzhang642

Assembly101 is a new procedural activity dataset featuring 4321 videos of people assembling and disassembling 101 "take-apart" toy vehicles. Participants work without fixed instructions, and the sequences feature rich and natural variations in action ordering, mistakes, and corrections. Assembly101 is the first multi-view action dataset, with simultaneous static (8) and egocentric (4) recordings. Sequences are annotated with more than 100K coarse and 1M fine-grained action segments, and 18M 3D hand poses. We benchmark on three action understanding tasks: recognition, anticipation and temporal segmentation. Additionally, we propose a novel task of detecting mistakes. The unique recording format and rich set of annotations allow us to investigate generalization to new toys, cross-view transfer, long-tailed distributions, and pose vs. appearance. We envision that Assembly101 will serve as a new challenge to investigate various activity understanding problems. Image Source: https://assembly-101.github.io/

Quora Question Pairs (QQP) dataset consists of over 400,000 question pairs, and each question pair is annotated with a binary value indicating whether the two questions are paraphrase of each other. Source: Bilateral Multi-Perspective Matching for Natural Language Sentences

UAV-Human is a large dataset for human behavior understanding with UAVs. It contains 67,428 multi-modal video sequences and 119 subjects for action recognition, 22,476 frames for pose estimation, 41,290 frames and 1,144 identities for person re-identification, and 22,263 frames for attribute recognition. The dataset was collected by a flying UAV in multiple urban and rural districts in both daytime and nighttime over three months, hence covering extensive diversities w.r.t subjects, backgrounds, illuminations, weathers, occlusions, camera motions, and UAV flying attitudes. This dataset can be used for UAV-based human behavior understanding, including action recognition, pose estimation, re-identification, and attribute recognition.

POP909 is a dataset which contains multiple versions of the piano arrangements of 909 popular songs created by professional musicians. The main body of the dataset contains the vocal melody, the lead instrument melody, and the piano accompaniment for each song in MIDI format, which are aligned to the original audio files. Furthermore, annotations are provided of tempo, beat, key, and chords, where the tempo curves are hand-labelled and others are done by MIR algorithms. Source: https://arxiv.org/pdf/2008.07142.pdf Image Source: https://arxiv.org/pdf/2008.07142.pdf

COCO-WholeBody is an extension of COCO dataset with whole-body annotations. There are 4 types of bounding boxes (person box, face box, left-hand box, and right-hand box) and 133 keypoints (17 for body, 6 for feet, 68 for face and 42 for hands) annotations for each person in the image. Source: Whole-Body Human Pose Estimation in the Wild Image source: https://arxiv.org/pdf/2007.11858v1.pdf

Ego4D is a massive-scale egocentric video dataset and benchmark suite. It offers 3,025 hours of daily life activity video spanning hundreds of scenarios (household, outdoor, workplace, leisure, etc.) captured by 855 unique camera wearers from 74 worldwide locations and 9 different countries. The approach to collection is designed to uphold rigorous privacy and ethics standards with consenting participants and robust de-identification procedures where relevant. Ego4D dramatically expands the volume of diverse egocentric video footage publicly available to the research community. Portions of the video are accompanied by audio, 3D meshes of the environment, eye gaze, stereo, and/or synchronized videos from multiple egocentric cameras at the same event. Furthermore, a host of new benchmark challenges are presented, centered around understanding the first-person visual experience in the past (querying an episodic memory), present (analyzing hand-object manipulation, audio-visual conversation, and social interactions), and future (forecasting activities). By publicly sharing this massive annotated dataset and benchmark suite, the aim is to push the frontier of first-person perception. Description from: Facebook AI Paper: Ego4D: Around the World in 3,000 Hours of Egocentric Video GitHub: https://github.com/EGO4D

A realistic dataset composed of 27 episodes from 6 popular TV series. The dataset spans over 16 hours of footage annotated with 30 action classes, totaling 6,231 action instances. Source: Online Action Detection

The Vid4 dataset is generally used for testing video super-resolution. It consists of four sequences: walk (740x480, 47 frames), foliage (740x480, 49 frames), city (704x576, 34 frames), and calendar (720x576, 41 frames).

Our dataset was made of videos from MSU Video Upscalers Benchmark Dataset, MSU Video Super-Resolution Benchmark Dataset and MSU Super-Resolution for Video Compression Benchmark Dataset. Dataset consists of real videos (were filmed with 2 cameras), video games footages, movies, cartoons, dynamic ads. How we brought our dataset closer to completeness? * The dataset covers a large number of use cases in the field of SR due to the large number of content types * The dataset contains videos with completely different resolutions, FPS values: 8, 24, 25, 30, 60, as well as high and low spatio-temporal complexity * Distorted videos were obtained using 46 SR methods, some of them were preprocessed with 5 codecs: aomenc, vvenc, x264, x265, uavs3es with different bitrates and qp values * The dataset was manually checked for redundancy Videos from benchmarks are FullHD video crops, since the subjective comparison was made on crops. Therefore, the resolution of all videos in the received dataset is low.

This is a dataset for a video super-resolution task. The dataset contains the most complex content for the restoration task: faces, text, QR-codes, car numbers, unpatterned textures, small details. Videos include different types of motion and different types of degradation: bicubic interpolation (BI) and Gaussian blurring and downsampling (BD). The resolution of all input video sequences is 480x320. Source: https://videoprocessing.ai/benchmarks/video-super-resolution.html Image Source: https://videoprocessing.ai/benchmarks/video-super-resolution.html

The dataset aims to find the algorithms that produce the most visually pleasant image possible and generalize well to a broad range of content. It consists of 30 clips and contains 15 2D-animated segments losslessly recorded from various video games and 15 camera-shot segments from high-bitrate YUV444 sources. The complexity of clips varies significantly in terms of spatial and temporal indexes. Multiple bicubic downscaling mixed with sharpening is used to simulate complex real-world camera degradation. The authors used slight compression and YUV420 conversion to simulate a practical use case. 1920×1080 sources were downscaled to 480×270 input.

We propose EMAGE, a framework to generate full-body human gestures from audio and masked gestures, encompassing facial, local body, hands, and global movements. To achieve this, we first introduce BEAT2 (BEAT-SMPLX-FLAME), a new mesh-level holistic co-speech dataset. BEAT2 combines MoShed SMPLX body with FLAME head parameters and further refines the modeling of head, neck, and finger movements, offering a community-standardized, high-quality 3D motion captured dataset. EMAGE leverages masked body gesture priors during training to boost inference performance. It involves a Masked Audio Gesture Transformer, facilitating joint training on audio-to-gesture generation and masked gesture reconstruction to effectively encode audio and body gesture hints. Encoded body hints from masked gestures are then separately employed to generate facial and body movements. Moreover, EMAGE adaptively merges speech features from the audio's rhythm and content and utilizes four compositional VQ-VAEs to enhance the results' fidelity and diversity. Experiments demonstrate that EMAGE generates holistic gestures with state-of-the-art performance and is flexible in accepting predefined spatial-temporal gesture inputs, generating complete, audio-synchronized results.

The iKala dataset is a singing voice separation dataset that comprises of 252 30-second excerpts sampled from 206 iKala songs (plus 100 hidden excerpts reserved for MIREX data mining contest). The music accompaniment and the singing voice are recorded at the left and right channels respectively. Additionally, the human-labeled pitch contours and timestamped lyrics are provided. This dataset is not available anymore. Source: http://mac.citi.sinica.edu.tw/ikala/ 1. "T" as a sofa: The "T" horizontal strip can mimic the back of a sofa with a delicate cushion or details of the uphols or appliances with the color button. The "T" vertical strip can show a feet or arm of the sofa, shiny, yet firm. 2. Merge "P": Put "P" next to "T", your curve to delicately with the top "T." It is intertwined. The circular part of "P" can show a cushion or a curved chair and synchronize the subject of furniture. Make sure "P" is visually relying on "T", which reflects the relationship of cohesion and balance. 5. Coherence of "B" and "I": "B" can be aligned as a pair of cushions or a modern chair, with mild curves with glossy and modern aesthetics. "I" can be a symbol of a shiny furniture or a vertical light bar and completes the shapes without overburdess them. Color palette 4: Includes soft soil colors such as beige, top and gray shades, along with silent or silver gold tips to touch elegance. Consider a slope effect to enhance modernity, to keep colors elegant and complex. 5. Connect the letters: Use the overlap or intertwined edges that the letters meet for the symbol of unity. The plan should allow viewers to distinguish each letter while feeling part of the same "structure". 6. Background patterns: Use delicate geometric patterns or textures that mimic fabrics or furniture materials such as wood seeds or woven fibers. These patterns must remain minimalist and focus on highlighting the logo, while maintaining communication. While it deals with the subject of furniture and design, this concept conveys modernity, creativity and professional. If you like, I can create a draft design for better visualization.

Touchdown is a corpus for executing navigation instructions and resolving spatial descriptions in visual real-world environments. The task is to follow instruction to a goal position and there find a hidden object, Touchdown the bear. Source: https://github.com/lil-lab/touchdown Image Source: https://github.com/lil-lab/touchdown

An annotated image memorability dataset to date (with 60,000 labeled images from a diverse array of sources). Source: Understanding and Predicting Image Memorability at a Large Scale

TextSeg is a large-scale fine-annotated and multi-purpose text detection and segmentation dataset, collecting scene and design text with six types of annotations: word- and character-wise bounding polygons, masks and transcriptions. Source: https://github.com/SHI-Labs/Rethinking-Text-Segmentation Image Source: https://github.com/SHI-Labs/Rethinking-Text-Segmentation

VideoLQ consists of videos downloaded from various video hosting sites such as Flickr and YouTube, with a Creative Common license.

AmazonQA consists of 923k questions, 3.6M answers and 14M reviews across 156k products. Building on the well-known Amazon dataset, additional annotations are collected, marking each question as either answerable or unanswerable based on the available reviews. Source: AmazonQA: A Review-Based Question Answering Task

ANTIQUE is a collection of 2,626 open-domain non-factoid questions from a diverse set of categories. The dataset contains 34,011 manual relevance annotations. The questions were asked by real users in a community question answering service, i.e., Yahoo! Answers. Relevance judgments for all the answers to each question were collected through crowdsourcing.

A video dataset for benchmarking upsampling methods. Inter4K contains 1,000 ultra-high resolution videos with 60 frames per second (fps) from online resources. The dataset provides standardized video resolutions at ultra-high definition (UHD/4K), quad-high definition (QHD/2K), full-high definition (FHD/1080p), (standard) high definition (HD/720p), one quarter of full HD (qHD/520p) and one ninth of a full HD (nHD/360p). We use frame rates of 60, 50, 30, 24 and 15 fps for each resolution. Based on this standardization, both super-resolution and frame interpolation tests can be performed for different scaling sizes ($\times 2$, $\times 3$ and $\times 4$). In this paper, we use Inter4K to address frame upsampling and interpolation. Inter4K provides both standardized UHD resolution and 60 fps for all of videos by also containing a diverse set of 1,000 5-second videos. Differences between scenes originate from the equipment (e.g., professional 4K cameras or phones), lighting conditions, variations in movements, actions or objects. The dataset is divided into 800 videos for training, 100 videos for validation and 100 videos for testing.

This is a dataset for a super-resolution task. The dataset contains 480x270 videos that were decoded with 6 different bitrates (100 - 4000 kbps) using 5 different codecs (H.264, H.265, H.266, AV1, and AVS3 standards). The dataset contains indoor and outdoor videos as well as animation. All videos have low SI/TI values and simple textures. It was made to minimize compression artifacts that may occur to make restoration of details possible.

EgoExoLearn is a fascinating dataset designed to bridge the gap between egocentric and exocentric views of procedural activities. 1. What Is EgoExoLearn? EgoExoLearn is a large-scale dataset that emulates how humans learn by observing others. It focuses on the process of asynchronous demonstration following. Participants in the dataset record egocentric videos as they perform tasks. These videos are guided by exocentric-view demonstration videos. In simpler terms, imagine someone watching a demonstration video (from an external perspective) and then replicating the same task while recording their own point-of-view video. 2. Dataset Details: EgoExoLearn dataset spans 120 hours and covers scenarios from daily life and specialized laboratories. It contains: - Egocentric videos: These are recorded by individuals executing tasks. - Demonstration videos: These show the same tasks from an external viewpoint. - Gaze data: High-quality gaze information accompanies the videos. - Multimodal annotations: Detailed annotations provide context and insights. 3. Applications and Benchmarks: The EgoExoLearn dataset serves as a playground for modeling the human ability and thus provides a playground to bridge asynchronous procedural actions from different viewpoints. It enables new benchmarks such as: - Cross-view association: Linking actions observed from different perspectives. - Cross-view action planning: Anticipating and planning actions based on both ego and exo views. - Cross-view referenced skill assessment: Evaluating skills across viewpoints. Using Exo-view demonstrations as guidance for better ego-view skill assessment. 4. Why Is It Important? - Understanding how we map others' activities into our own point of view is a fundamental human skill. - EgoExoLearn paves the way for creating AI agents capable of seamlessly learning by observing humans in the real world. (1) EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric .... https://arxiv.org/html/2403.16182v1. (2) EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric .... https://egoexolearn.github.io/. (3) EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric .... https://arxiv.org/abs/2403.16182. (4) EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric .... https://allainews.com/item/egoexolearn-a-dataset-for-bridging-asynchronous-ego-and-exo-centric-view-of-procedural-activities-in-real-world-2024-03-26/. (5) undefined. https://github.com/OpenGVLab/EgoExoLearn/.

A new dataset of handwritten text with fine-grained annotations at the character level and report results from an initial user evaluation. Source: DeepWriting: Making Digital Ink Editable via Deep Generative Modeling