Domain Generalization

Cityscapes is a large-scale database which focuses on semantic understanding of urban street scenes. It provides semantic, instance-wise, and dense pixel annotations for 30 classes grouped into 8 categories (flat surfaces, humans, vehicles, constructions, objects, nature, sky, and void). The dataset consists of around 5000 fine annotated images and 20000 coarse annotated ones. Data was captured in 50 cities during several months, daytimes, and good weather conditions. It was originally recorded as video so the frames were manually selected to have the following features: large number of dynamic objects, varying scene layout, and varying background. Source: A Review on Deep Learning Techniques Applied to Semantic Segmentation Image Source: https://www.cityscapes-dataset.com/dataset-overview/

Fashion-MNIST is a dataset comprising of 28×28 grayscale images of 70,000 fashion products from 10 categories, with 7,000 images per category. The training set has 60,000 images and the test set has 10,000 images. Fashion-MNIST shares the same image size, data format and the structure of training and testing splits with the original MNIST. Source: Generative Probabilistic Novelty Detection with Adversarial Autoencoders Image Source: https://github.com/zalandoresearch/fashion-mnist

Office-Home is a benchmark dataset for domain adaptation which contains 4 domains where each domain consists of 65 categories. The four domains are: Art – artistic images in the form of sketches, paintings, ornamentation, etc.; Clipart – collection of clipart images; Product – images of objects without a background and Real-World – images of objects captured with a regular camera. It contains 15,500 images, with an average of around 70 images per class and a maximum of 99 images in a class. Source: Multi-component Image Translation for Deep Domain Generalization Image Source: Wen et al

DomainNet is a dataset of common objects in six different domain. All domains include 345 categories (classes) of objects such as Bracelet, plane, bird and cello. The domains include clipart: collection of clipart images; real: photos and real world images; sketch: sketches of specific objects; infograph: infographic images with specific object; painting artistic depictions of objects in the form of paintings and quickdraw: drawings of the worldwide players of game “Quick Draw!”. Source: What is being transferred in transfer learning? Image Source: http://ai.bu.edu/M3SDA/

PACS is an image dataset for domain generalization. It consists of four domains, namely Photo (1,670 images), Art Painting (2,048 images), Cartoon (2,344 images) and Sketch (3,929 images). Each domain contains seven categories. Source: Deep Domain-Adversarial Image Generation for Domain Generalisation Image Source: https://www.researchgate.net/figure/Sample-images-from-PACS-dataset-Each-row-represents-a-domain-and-each-column-represents_fig1_334695033

ImageNet-C is an open source data set that consists of algorithmically generated corruptions (blur, noise) applied to the ImageNet test-set. Source: Selective Brain Damage: Measuring the Disparate Impact of Model Pruning Image Source: https://arxiv.org/pdf/1807.01697.pdf

Common corruptions dataset for CIFAR10

ImageNet-R(endition) contains art, cartoons, deviantart, graffiti, embroidery, graphics, origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, and video game renditions of ImageNet classes. ImageNet-R has renditions of 200 ImageNet classes resulting in 30,000 images. Source: The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

The ImageNet-A dataset consists of real-world, unmodified, and naturally occurring examples that are misclassified by ResNet models. Source: On Robustness and Transferability of Convolutional Neural Networks Image Source: https://github.com/hendrycks/natural-adv-examples

ImageNet-Sketch data set consists of 50,889 images, approximately 50 images for each of the 1000 ImageNet classes. The data set is constructed with Google Image queries "sketch of __", where __ is the standard class name. Only within the "black and white" color scheme is searched. 100 images are initially queried for every class, and the pulled images are cleaned by deleting the irrelevant images and images that are for similar but different classes. For some classes, there are less than 50 images after manually cleaning, and then the data set is augmented by flipping and rotating the images. Source: ImageNet-Sketch Image Source: https://github.com/HaohanWang/ImageNet-Sketch

The 'shape bias' dataset was introduced in Geirhos et al. (ICLR 2019) and consists of 224x224 images with conflicting texture and shape information (e.g., cat shape with elephant texture). This is used to measure the shape vs. texture bias of image classifiers.

The Stylized-ImageNet dataset is created by removing local texture cues in ImageNet while retaining global shape information on natural images via AdaIN style transfer. This nudges CNNs towards learning more about shapes and less about local textures. Source: Adversarial Examples Improve Image Recognition Image Source: https://github.com/rgeirhos/Stylized-ImageNet

WildDash is a benchmark evaluation method is presented that uses the meta-information to calculate the robustness of a given algorithm with respect to the individual hazards. Source: WildDash - Creating Hazard-Aware Benchmarks Image Source: https://wilddash.cc/

VLCS is a dataset to test for domain generalization.

ImageNet-P consists of noise, blur, weather, and digital distortions. The dataset has validation perturbations; has difficulty levels; has CIFAR-10, Tiny ImageNet, ImageNet 64 × 64, standard, and Inception-sized editions; and has been designed for benchmarking not training networks. ImageNet-P departs from ImageNet-C by having perturbation sequences generated from each ImageNet validation image. Each sequence contains more than 30 frames, so to counteract an increase in dataset size and evaluation time only 10 common perturbations are used. Source: Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

The goal of NICO Challenge is to facilitate the OOD (Out-of-Distribution) generalization in visual recognition through promoting the research on the intrinsic learning mechanisms with native invariance and generalization ability. The training data is a mixture of several observed contexts while the test data is composed of unseen contexts. Participants are tasked with developing reliable algorithms across different contexts (domains) to improve the generalization ability of models.

We introduce ACDC, the Adverse Conditions Dataset with Correspondences for training and testing semantic segmentation methods on adverse visual conditions. It comprises a large set of 4006 images which are evenly distributed between fog, nighttime, rain, and snow. Each adverse-condition image comes with a high-quality fine pixel-level semantic annotation, a corresponding image of the same scene taken under normal conditions and a binary mask that distinguishes between intra-image regions of clear and uncertain semantic content. ACDC supports two tasks: 1. standard semantic segmentation 2. uncertainty-aware semantic segmentation

Our goal is to improve upon the status quo for designing image classification models trained in one domain that perform well on images from another domain. Complementing existing work in robustness testing, we introduce the first test dataset for this purpose which comes from an authentic use case where photographers wanted to learn about the content in their images. We built a new test set using 8,900 images taken by people who are blind for which we collected metadata to indicate the presence versus absence of 200 ImageNet object categories. We call this dataset VizWiz-Classification.

The MSK dataset is a dataset for lesion recognition from the Memorial Sloan-Kettering Cancer Center. It is used as part of the ISIC lesion recognition challenges. Source: https://arxiv.org/pdf/1710.05006.pdf Image Source: https://arxiv.org/pdf/1902.03368.pdf

Super-CLEVR is a dataset for Visual Question Answering (VQA) where different factors in VQA domain shifts can be isolated in order that their effects can be studied independently. It contains 21 vehicle models belonging to 5 categories, with controllable attributes. Four factors are considered: visual complexity, question redundancy, concept distribution and concept compositionality. Source: Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning Image Source: https://arxiv.org/pdf/2212.00259v1.pdf

I.I.D. hypothesis between training and testing data is the basis of numerous image classification methods. Such property can hardly be guaranteed in practice where the Non-IIDness is common, causing in- stable performances of these models. In literature, however, the Non-I.I.D. image classification problem is largely understudied. A key reason is lacking of a well-designed dataset to support related research. In this paper, we construct and release a Non-I.I.D. image dataset called NICO, which uses contexts to create Non-IIDness consciously. Compared to other datasets, extended analyses prove NICO can support various Non-I.I.D. situations with sufficient flexibility. Meanwhile, we propose a baseline model with Con- vNet structure for General Non-I.I.D. image classification, where distribution of testing data is unknown but different from training data. The experimental results demonstrate that NICO can well support the training of ConvNet model from scratch, and a batch balancing module can help ConvNets to perform better in Non-I.I.D. settings.

Wild-Time is a benchmark of 5 datasets that reflect temporal distribution shifts arising in a variety of real-world applications, including patient prognosis and news classification. On these datasets, we systematically benchmark 13 prior approaches, including methods in domain generalization, continual learning, self-supervised learning, and ensemble learning. Source: Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time Image Source: https://arxiv.org/pdf/2211.14238v1.pdf

• The Sims4Action Dataset: a videogame-based dataset for Synthetic→Real domain adaptation for human activity recognition. * Goal : Exploring the concept of constructing training examples for Activities of Daily Living (ADL) recognition by playing life simulation video games. * ** Sims4Action dataset is created with the commercial game THE SIMS 4** by executing actions-of-interest within the game in a "top-down" manner. It features ten hours of video material of eight diverse characters and multiple environments. Ten actions are selected to have a direct correspondence to categories covered in the real-life dataset Toyota Smarthome [2] to enable the research of Synthetic→Real transfer in action recognition. * Two benchmarks :* Gaming→Gaming* (training and evaluation on Sims4Action) and Gaming→Real (training on Sims4Action, evaluation on the real Toyota Smarthome data [2]). * Main challenge: Gaming→Real domain adaptation While ADL recognition on gaming data is interesting from a theoretical perspective, the key challenge arises from transferring knowledge learned from simulated data to real-world applications. Sims4Action specifically provides a benchmark for this scenario since it describes a Gaming→Real challenge, which evaluates models on real videos derived from the existing Toyota Smarthome dataset . # References [1] Let's Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Games. Alina Roitberg*, David Schneider*, Aulia Djamal, Constantin Seibold, Simon Reiß, Rainer Stiefelhagen, In International Conference on Intelligent Robots and Systems (IROS), 2021 (* denotes equal contribution.) [2] Toyota smarthome: Real-world activities of daily living. Srijan Das, Rui Dai, Michal Koperski, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, Gianpiero Francesca, In International Conference on Computer Vision (ICCV), 2019.

As part of an ongoing worldwide effort to comprehend and monitor insect biodiversity, we present the BIOSCAN-5M Insect dataset to the machine learning community. BIOSCAN-5M is a comprehensive dataset containing multi-modal information for over 5 million insect specimens, and it significantly expands existing image-based biological datasets by including taxonomic labels, raw nucleotide barcode sequences, assigned barcode index numbers, geographical information, and specimen size. Every record has both image and DNA data. Each record of the BIOSCAN-5M dataset contains six primary attributes: RGB image DNA barcode sequence Barcode Index Number (BIN) Biological taxonomic classification Geographical information Specimen size

Human-Animal-Cartoon (HAC) dataset consists of seven actions (‘sleeping’, ‘watching tv’, ‘eating’, ‘drinking’, ‘swimming’, ‘running’, and ‘opening door’) performed by humans, animals, and cartoon figures, forming three different domains. 3381 video clips are collected from the internet with around 1000 for each domain and three modalities are provided in the dataset: video, audio, and optical flow.

We design an all-day semantic segmentation benchmark all-day CityScapes. It is the first semantic segmentation benchmark that contains samples from all-day scenarios, i.e., from dawn to night. Our dataset will be made publicly available at [https://isis-data.science.uva.nl/cv/1ADcityscape.zip].

Caltech Fish Counting Dataset (CFC) is a large-scale dataset for detecting, tracking, and counting fish in sonar videos. This dataset contains over 1,500 videos sourced from seven different sonar cameras.

This prostate MRI segmentation dataset is collected from six different data sources. Source: https://github.com/liuquande/SAML

In our benchmark WHYSHIFT, we explore distribution shifts on 5 real-world tabular datasets from the economic and traffic sectors with natural spatiotemporal distribution shifts.We only pick 7 typical settings out of 22 settings and select only one representative target domain for each setting. In our benchmark, we specify the distribution shift pattern for each setting, and we provide the tools to identify risky regions with large $Y|X$ shifts and to diagnose the performance degradation.

An adaption of the MVTec Anomaly Detection dataset, presented in the paper "Domain-independent detection of known anomalies". There are three different datasets, each covering one specific anomaly type: color, cut and hole. The datasets can be used to evaluate approaches on the hybrid task of detecting known anomalies across different, previously unseen objects: All object types except one are used for training. During testing, the images of the remaining object type should be classified on whether they contain an anomaly. Note: the authors are not affiliated with MVTec

The Tufts fNIRS to Mental Workload (fNIRS2MW) open-access dataset is a new dataset for building machine learning classifiers that can consume a short window (30 seconds) of multivariate fNIRS recordings and predict the mental workload intensity of the user during that window. You can use this dataset for tasks like - time series classification using sliding windows - domain adaptation or domain generalization (how well does your classifier generalize to a new subject?) - fairness of time series classifiers (does performance of your classifier vary by subject race or gender?) Useful Links: * Project Website (and data download links): https://tufts-hci-lab.github.io/code_and_datasets/fNIRS2MW.html * Code for benchmarks: https://github.com/tufts-ml/fNIRS-mental-workload-classifiers * DataSheet documentation: https://github.com/tufts-ml/fNIRS-mental-workload-classifiers/blob/main/Datasheet-Tufts-fNIRS2MW.pdf * Academic Paper (published at NeurIPS Datasets & Benchmarks '21) https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/bd686fd640be98efaae0091fa301e613-Paper-round2.pdf ## Motivation We are interested in building brain computer interfaces (BCIs) that would help out everyday computer users working at a desktop or laptop. In our target future use case, a user would actively use a keyboard and mouse as usual, but also wear a non-intrusive headband sensor that would passively provide real-time measurements of brain activity to the computer. Based on moment-to-moment estimates of mental workload, the computer could adjust the interface to support the user. Functional near-infrared spectroscopy (fNIRS) is a promising sensor technology for achieving this goal of "everyday BCI", compared to alternatives like EEG or fMRI. We have developed a prototype fNIRS probe mounted on a headband that we used to collect this dataset (see our paper for details). # Dataset Overview For a complete dataset summary, see our public DataSheet PDF For each participant (68 recommended; 87 total), the dataset contains the following records obtained during one 30-60 minute experimental session. Each subject contributes just over 21 minutes of fNIRS data from the desired n-back experimental conditions, with remaining time related to rest or instruction periods. - fNIRS recordings - Multivariate (D=8) time-series representing brain activity throughout the session, recorded by a sensor probe placed on the forehead and secured via headband - All measurements are recorded at a regular sampling rate of 5.2 Hz. - At each timestep, we record 8 real-valued measurements, one for each combination of - 2 blood chemical concentration changes (oxygenated hemoglobin and deoxygenated hemoglobin) - 2 optical data types used for the measurement (intensity and phase) - 2 spatial locations on the forehead. - The units of each measurement are micro-moles of (oxy-/deoxy-)hemoglobin per liter of tissue. - Activity labels - Annotations of the experimental task activity the subject performed throughout the session, including instruction, rest, and active experiment segments. - We label each segment of the active experiment as one of four possible n-back working memory intensity levels (0-back, 1-back, 2-back, or 3-back). Increased intensity levels are intended to induce an increased level of cognitive workload. - For all experiments reported in the paper, we focus on a binary task (0 vs 2 back) - Demographics - The participant’s age, gender, race, handedness, and other attributes. This lets us measure and audit performance by subpopulation (e.g. how does the classifier perform on white subjects vs. black subjects). ## Publications The Tufts fNIRS Mental Workload Dataset & Benchmark for Brain-Computer Interfaces that Generalize Zhe Huang, Liang Wang, Giles Blaney, Christopher Slaughter, Devon McKeon, Ziyu Zhou, Robert Jacob, and Michael C. Hughes To appear in the Proceedings of Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks , 2021 Link to Paper PDF: <a href="https://openreview.net/pdf?id=QzNHE7QHhut">https://openreview.net/pdf?id=QzNHE7QHhut</a>