Image Generation

DensePose-COCO is a large-scale ground-truth dataset with image-to-surface correspondences manually annotated on 50K COCO images and train DensePose-RCNN, to densely regress part-specific UV coordinates within every human region at multiple frames per second. Source: DensePose: Dense Human Pose Estimation In The Wildc Image Source: Guler et al

EMNIST (extended MNIST) has 4 times more data than MNIST. It is a set of handwritten digits with a 28 x 28 format. Source: Domain Discrepancy Measure for Complex Models in Unsupervised Domain Adaptation Image Source: https://arxiv.org/pdf/1803.01900.pdf

Foggy Cityscapes is a synthetic foggy dataset which simulates fog on real scenes. Each foggy image is rendered with a clear image and depth map from Cityscapes. Thus the annotations and data split in Foggy Cityscapes are inherited from Cityscapes. Source: Exploring Object Relation in Mean Teacher for Cross-Domain Detection Image Source: http://people.ee.ethz.ch/~csakarid/SFSU_synthetic/

MIMIC-CXR from Massachusetts Institute of Technology presents 371,920 chest X-rays associated with 227,943 imaging studies from 65,079 patients. The studies were performed at Beth Israel Deaconess Medical Center in Boston, MA. Source: Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification Image Source: https://arxiv.org/abs/1901.07042

ImageNet Long-Tailed is a subset of /dataset/imagenet dataset consisting of 115.8K images from 1000 categories, with maximally 1280 images per class and minimally 5 images per class. The additional classes of images in ImageNet-2010 are used as the open set. Source: Large-Scale Long-Tailed Recognition in an Open World

The PASCAL Visual Object Classes (VOC) 2012 dataset contains 20 object categories including vehicles, household, animals, and other: aeroplane, bicycle, boat, bus, car, motorbike, train, bottle, chair, dining table, potted plant, sofa, TV/monitor, bird, cat, cow, dog, horse, sheep, and person. Each image in this dataset has pixel-level segmentation annotations, bounding box annotations, and object class annotations. This dataset has been widely used as a benchmark for object detection, semantic segmentation, and classification tasks. The PASCAL VOC dataset is split into three subsets: 1,464 images for training, 1,449 images for validation and a private testing set. Source: Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey Image Source: http://host.robots.ox.ac.uk/pascal/VOC/voc2012/examples/images/sheep_06.jpg

The Lip Reading in the Wild (LRW) dataset a large-scale audio-visual database that contains 500 different words from over 1,000 speakers. Each utterance has 29 frames, whose boundary is centered around the target word. The database is divided into training, validation and test sets. The training set contains at least 800 utterances for each class while the validation and test sets contain 50 utterances. Source: Towards Pose-invariant Lip-Reading Image Source: https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrw1.html

CelebAMask-HQ is a large-scale face image dataset that has 30,000 high-resolution face images selected from the CelebA dataset by following CelebA-HQ. Each image has segmentation mask of facial attributes corresponding to CelebA. Source: https://github.com/switchablenorms/CelebAMask-HQ Image Source: https://github.com/switchablenorms/CelebAMask-HQ

ViZDoom is an AI research platform based on the classical First Person Shooter game Doom. The most popular game mode is probably the so-called Death Match, where several players join in a maze and fight against each other. After a fixed time, the match ends and all the players are ranked by the FRAG scores defined as kills minus suicides. During the game, each player can access various observations, including the first-person view screen pixels, the corresponding depth-map and segmentation-map (pixel-wise object labels), the bird-view maze map, etc. The valid actions include almost all the keyboard-stroke and mouse-control a human player can take, accounting for moving, turning, jumping, shooting, changing weapon, etc. ViZDoom can run a game either synchronously or asynchronously, indicating whether the game core waits until all players’ actions are collected or runs in a constant frame rate without waiting. Source: Arena: a toolkit for Multi-Agent Reinforcement Learning Image Source: https://github.com/mwydmuch/ViZDoom

• Visible-infrared Paired Dataset for Low-light Vision * 30976 images (15488 pairs) * 24 dark scenes, 2 daytime scenes * Support for image-to-image translation (visible to infrared, or infrared to visible), visible and infrared image fusion, low-light pedestrian detection, and infrared pedestrian detection * (The original image and video pairs (before registration) of LLVIP are also released!)

The Oxford-BBC Lip Reading Sentences 2 (LRS2) dataset is one of the largest publicly available datasets for lip reading sentences in-the-wild. The database consists of mainly news and talk shows from BBC programs. Each sentence is up to 100 characters in length. The training, validation and test sets are divided according to broadcast date. It is a challenging set since it contains thousands of speakers without speaker labels and large variation in head pose. The pre-training set contains 96,318 utterances, the training set contains 45,839 utterances, the validation set contains 1,082 utterances and the test set contains 1,242 utterances. Source: Audio-visual Recognition of Overlapped speech for the LRS2 dataset Image Source: https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrs2.html

Recent breakthroughs in diffusion models, multimodal pretraining, and efficient finetuning have led to an explosion of text-to-image generative models. Given human evaluation is expensive and difficult to scale, automated methods are critical for evaluating the increasingly large number of new models. However, most current automated evaluation metrics like FID or CLIPScore only offer a holistic measure of image quality or image-text alignment, and are unsuited for fine-grained or instance-level analysis. In this paper, we introduce GenEval, an object-focused framework to evaluate compositional image properties such as object co-occurrence, position, count, and color. We show that current object detection models can be leveraged to evaluate text-to-image models on a variety of generation tasks with strong human agreement, and that other discriminative vision models can be linked to this pipeline to further verify properties like object color. We then evaluate several open-source text-to-image models and analyze their relative generative capabilities on our benchmark. We find that recent models demonstrate significant improvement on these tasks, though they are still lacking in complex capabilities such as spatial relations and attribute binding. Finally, we demonstrate how GenEval might be used to help discover existing failure modes, in order to inform development of the next generation of text-to-image models. Our code to run the GenEval framework is publicly available at this https URL.

Structured3D is a large-scale photo-realistic dataset containing 3.5K house designs (a) created by professional designers with a variety of ground truth 3D structure annotations (b) and generate photo-realistic 2D images (c). The dataset consists of rendering images and corresponding ground truth annotations (e.g., semantic, albedo, depth, surface normal, layout) under different lighting and furniture configurations. Source: https://github.com/bertjiazheng/Structured3D Image Source: https://github.com/bertjiazheng/Structured3D

VITON-HD dataset is a dataset for high-resolution (i.e., 1024x768) virtual try-on of clothing items. Specifically, it consists of 13,679 frontal-view woman and top clothing image pairs.

DrawBench is a comprehensive and challenging benchmark for text-to-image models, introduced by the Imagen research team. Let me provide you with more details: 1. Purpose and Context: - DrawBench serves as an evaluation benchmark specifically designed to assess the performance of text-to-image models. - It allows researchers and practitioners to compare different methods and understand their strengths and weaknesses in generating images from textual descriptions. 2. Imagen: Text-to-Image Diffusion Models: - Imagen is a state-of-the-art text-to-image diffusion model developed by the Google Research Brain Team. - It combines the power of large transformer language models (such as T5) for understanding text with the strength of diffusion models for high-fidelity image generation. - Key Discovery: Imagen demonstrates that generic large language models pretrained on text-only corpora are remarkably effective at encoding text for image synthesis. - Photorealism and Language Understanding: Imagen achieves an unprecedented degree of photorealism and a deep level of language understanding. - FID Score: It achieves a new state-of-the-art FID (Fréchet Inception Distance) score of 7.27 on the COCO dataset, without ever being trained on COCO. - Human Raters' Perception: Human raters find Imagen samples to be on par with the COCO data itself in terms of image-text alignment. 3. DrawBench: A Comprehensive Benchmark: - DrawBench provides a rigorous evaluation framework for text-to-image models. - Researchers can compare Imagen with other recent methods, including VQ-GAN+CLIP, Latent Diffusion Models, and DALL-E 2. - Human raters prefer Imagen over other models in side-by-side comparisons, considering both sample quality and image-text alignment. 4. Examples from the Imagen Family: - Imagen generates diverse and imaginative images based on textual prompts. Here are some examples: - A strawberry mug filled with white sesame seeds, floating in a dark chocolate sea. - A brain riding a rocketship heading towards the moon. - A dragon fruit wearing a karate belt in the snow. - A small cactus wearing a straw hat and neon sunglasses in the Sahara desert. - A photo of a Corgi dog riding a bike in Times Square, wearing sunglasses and a beach hat. - Teddy bears swimming at the Olympics 400m Butterfly event. - Sprouts in the shape of the text 'Imagen' coming out of a fairytale book. - A transparent sculpture of a duck made out of glass, in front of a painting of a landscape. - A single beam of light entering the room from the ceiling, illuminating an easel with a Rembrandt painting of a raccoon. 5. Technical Details: - Imagen uses a large frozen T5-XXL encoder to encode input text into embeddings. - The combination of language understanding and diffusion-based image generation results in high-quality, contextually relevant images. Source: Conversation with Bing, 3/18/2024 (1) Imagen: Text-to-Image Diffusion Models. https://imagen.research.google/. (2) Evaluating Diffusion Models - Hugging Face. https://huggingface.co/docs/diffusers/conceptual/evaluation. (3) shunk031/DrawBench · Datasets at Hugging Face. https://huggingface.co/datasets/shunk031/DrawBench. (4) sayakpaul/drawbench · Datasets at Hugging Face. https://huggingface.co/datasets/sayakpaul/drawbench.

The Radboud Faces Database (RaFD) is a set of pictures of 67 models (both adult and children, males and females) displaying 8 emotional expressions. Source: http://www.socsci.ru.nl:8180/RaFD2/RaFD Image Source: http://www.socsci.ru.nl:8180/RaFD2/RaFD

ARKitScenes is an RGB-D dataset captured with the widely available Apple LiDAR scanner. Along with the per-frame raw data (Wide Camera RGB, Ultra Wide camera RGB, LiDar scanner depth, IMU) the authors also provide the estimated ARKit camera pose and ARKit scene reconstruction for each iPad Pro sequence. In addition to the raw and processed data from the mobile device, ARKit.

MetFaces is an image dataset of human faces extracted from works of art. The dataset consists of 1336 high-quality PNG images at 1024×1024 resolution. The images were downloaded via the Metropolitan Museum of Art Collection API, and automatically aligned and cropped using dlib. Various automatic filters were used to prune the set.

ApolloScape is a large dataset consisting of over 140,000 video frames (73 street scene videos) from various locations in China under varying weather conditions. Pixel-wise semantic annotation of the recorded data is provided in 2D, with point-wise semantic annotation in 3D for 28 classes. In addition, the dataset contains lane marking annotations in 2D. Source: A2D2: Audi Autonomous Driving Dataset Image Source: https://arxiv.org/pdf/1803.06184.pdf

Recipe1M+ is a dataset which contains one million structured cooking recipes with 13M associated images. Source: Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images Image Source: http://im2recipe.csail.mit.edu/

T2I-CompBench is a comprehensive benchmark for open-world compositional text-to-image generation, consisting of 6,000 compositional textual prompts from 3 categories (attribute binding, object relationships, and complex compositions) and 6 sub-categories (color binding, shape binding, texture binding, spatial relationships, non-spatial relationships, and complex compositions).

The Places365 dataset is a scene recognition dataset. It is composed of 10 million images comprising 434 scene classes. There are two versions of the dataset: Places365-Standard with 1.8 million train and 36000 validation images from K=365 scene classes, and Places365-Challenge-2016, in which the size of the training set is increased up to 6.2 million extra images, including 69 new scene classes (leading to a total of 8 million train images from 434 scene classes). Source: Semantic-Aware Scene Recognition Image Source: Places

VOCASET is a 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio. The dataset has 12 subjects and 480 sequences of about 3-4 seconds each with sentences chosen from an array of standard protocols that maximize phonetic diversity. Source: timzhang642

The Stanford Dogs dataset contains 20,580 images of 120 classes of dogs from around the world, which are divided into 12,000 images for training and 8,580 images for testing. Source: Universal-to-Specific Framework for Complex Action Recognition Image Source: https://www.tensorflow.org/datasets/catalog/stanford_dogs

Vision and Language Navigation in Continuous Environments (VLN-CE) is an instruction-guided navigation task with crowdsourced instructions, realistic environments, and unconstrained agent navigation. The dataset consists of 4475 trajectories converted from Room-to-Room train and validation splits. For each trajectory, multiple natural language instructions from Room-to-Room and a pre-computed shortest path are provided following the waypoints via low-level actions.

Pick-a-Pic dataset was created by logging user interactions with the Pick-a-Pic web application for text-to image generation. Overall, the Pick-a-Pic dataset contains over 500,000 examples and 35,000 distinct prompts. Each example contains a prompt, two generated images, and a label for which image is preferred, or if there is a tie when no image is significantly preferred over the other. Source: Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation Image Source: Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation

Synscapes is a synthetic dataset for street scene parsing created using photorealistic rendering techniques, and show state-of-the-art results for training and validation as well as new types of analysis. Source: Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing Image Source: https://7dlabs.com/synscapes-overview

The Stacked MNIST dataset is derived from the standard MNIST dataset with an increased number of discrete modes. 240,000 RGB images in the size of 32×32 are synthesized by stacking three random digit images from MNIST along the color channel, resulting in 1,000 explicit modes in a uniform distribution corresponding to the number of possible triples of digits. Source: Inclusive GAN: Improving Data and Minority Coverage in Generative Models Image Source: https://arxiv.org/abs/1705.07761

iHarmony4 is a synthesized dataset for Image Harmonization. It contains 4 sub-datasets: HCOCO, HAdobe5k, HFlickr, and Hday2night (based on COCO, Adobe5k, Flickr, day2night datasets respectively), each of which contains synthesized composite images, foreground masks of composite images and corresponding real images. Source: Image_Harmonization_Datasets

Fashion-Gen consists of 293,008 high definition (1360 x 1360 pixels) fashion images paired with item descriptions provided by professional stylists. Each item is photographed from a variety of angles. Source: Fashion-Gen: The Generative Fashion Dataset and Challenge Image Source: Rostamzadeh et al

The General Video Game AI (GVGAI) framework is widely used in research which features a corpus of over 100 single-player games and 60 two-player games. These are fairly small games, each focusing on specific mechanics or skills the players should be able to demonstrate, including clones of classic arcade games such as Space Invaders, puzzle games like Sokoban, adventure games like Zelda or game-theory problems such as the Iterative Prisoners Dilemma. All games are real-time and require players to make decisions in only 40ms at every game tick, although not all games explicitly reward or require fast reactions; in fact, some of the best game-playing approaches add up the time in the beginning of the game to run Breadth-First Search in puzzle games in order to find an accurate solution. However, given the large variety of games (many of which are stochastic and difficult to predict accurately), scoring systems and termination conditions, all unknown to the players, highly-adaptive general methods are needed to tackle the diverse challenges proposed. Source: Rolling Horizon Evolutionary Algorithms for General Video Game Playing Image Source: http://www.gvgai.net/

Enables detailed human body model reconstruction in clothing from a single monocular RGB video without requiring a pre scanned template or manually clicked points. Source: Video Based Reconstruction of 3D People Models

UT Zappos50K is a large shoe dataset consisting of 50,025 catalog images collected from Zappos.com. The images are divided into 4 major categories — shoes, sandals, slippers, and boots — followed by functional types and individual brands. The shoes are centered on a white background and pictured in the same orientation for convenient analysis. Source: UT Zappos50K

CREMA-D is an emotional multimodal actor data set of 7,442 original clips from 91 actors. These clips were from 48 male and 43 female actors between the ages of 20 and 74 coming from a variety of races and ethnicities (African America, Asian, Caucasian, Hispanic, and Unspecified). Actors spoke from a selection of 12 sentences. The sentences were presented using one of six different emotions (Anger, Disgust, Fear, Happy, Neutral, and Sad) and four different emotion levels (Low, Medium, High, and Unspecified). Participants rated the emotion and emotion levels based on the combined audiovisual presentation, the video alone, and the audio alone. Due to the large number of ratings needed, this effort was crowd-sourced and a total of 2443 participants each rated 90 unique clips, 30 audio, 30 visual, and 30 audio-visual. 95% of the clips have more than 7 ratings.

A dataset of 90,000 high-resolution nature landscape images, crawled from Unsplash and Flickr and preprocessed with Mask R-CNN and Inception V3.