Out-of-Distribution Detection

The CIFAR-100 dataset (Canadian Institute for Advanced Research, 100 classes) is a subset of the Tiny Images dataset and consists of 60000 32x32 color images. The 100 classes in the CIFAR-100 are grouped into 20 superclasses. There are 600 images per class. Each image comes with a "fine" label (the class to which it belongs) and a "coarse" label (the superclass to which it belongs). There are 500 training images and 100 testing images per class. The criteria for deciding whether an image belongs to a class were as follows: * The class name should be high on the list of likely answers to the question “What is in this picture?” * The image should be photo-realistic. Labelers were instructed to reject line drawings. * The image should contain only one prominent instance of the object to which the class refers. * The object may be partially occluded or seen from an unusual viewpoint as long as its identity is still clear to the labeler. Source: https://www.cs.toronto.edu/~kriz/cifar.html Image Source: https://www.cs.toronto.edu/~kriz/cifar.html

Fashion-MNIST is a dataset comprising of 28×28 grayscale images of 70,000 fashion products from 10 categories, with 7,000 images per category. The training set has 60,000 images and the test set has 10,000 images. Fashion-MNIST shares the same image size, data format and the structure of training and testing splits with the original MNIST. Source: Generative Probabilistic Novelty Detection with Adversarial Autoencoders Image Source: https://github.com/zalandoresearch/fashion-mnist

The Stanford Sentiment Treebank is a corpus with fully labeled parse trees that allows for a complete analysis of the compositional effects of sentiment in language. The corpus is based on the dataset introduced by Pang and Lee (2005) and consists of 11,855 single sentences extracted from movie reviews. It was parsed with the Stanford parser and includes a total of 215,154 unique phrases from those parse trees, each annotated by 3 human judges. Each phrase is labelled as either negative, somewhat negative, neutral, somewhat positive or positive. The corpus with all 5 labels is referred to as SST-5 or SST fine-grained. Binary classification experiments on full sentences (negative or somewhat negative vs somewhat positive or positive with neutral sentences discarded) refer to the dataset as SST-2 or SST binary.

The STL-10 is an image dataset derived from ImageNet and popularly used to evaluate algorithms of unsupervised feature learning or self-taught learning. Besides 100,000 unlabeled images, it contains 13,000 labeled images from 10 object classes (such as birds, cats, trucks), among which 5,000 images are partitioned for training while the remaining 8,000 images for testing. All the images are color images with 96×96 pixels in size. Source: Unsupervised Feature Learning with C-SVDDNet Image Source: https://cs.stanford.edu/~acoates/stl10/

iSUN is a ground truth of gaze traces on images from the SUN dataset. The collection is partitioned into 6,000 images for training, 926 for validation and 2,000 for test. Source: End-to-end Convolutional Network for Saliency Prediction Image Source: http://turkergaze.cs.princeton.edu/

The Places365 dataset is a scene recognition dataset. It is composed of 10 million images comprising 434 scene classes. There are two versions of the dataset: Places365-Standard with 1.8 million train and 36000 validation images from K=365 scene classes, and Places365-Challenge-2016, in which the size of the training set is increased up to 6.2 million extra images, including 69 new scene classes (leading to a total of 8 million train images from 434 scene classes). Source: Semantic-Aware Scene Recognition Image Source: Places

It is manually annotated, comes with a naturally diverse distribution, and has a large scale. It is built to overcome several shortcomings of existing OOD benchmarks. OpenImage-O is image-by-image filtered from the test set of OpenImage-V3, which has been collected from Flickr without a predefined list of class names or tags, leading to natural class statistics and avoiding an initial design bias.

The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups.

A benchmark dataset for out-of-distribution detection. ImageNet-1k is in-distribution, while Textures is out-of-distribution.

A benchmark dataset for out-of-distribution detection. ImageNet-1k is in-distribution, while iNaturalist is out-of-distribution.

A benchmark dataset for out-of-distribution detection. ImageNet-1k is in-distribution, while Places is out-of-distribution.

A benchmark dataset for out-of-distribution detection. ImageNet-1k is in-distribution, while SUN is out-of-distribution.

The NINCO (No ImageNet Class Objects) dataset is introduced in the ICML 2023 paper In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation. The images in this dataset are free from objects that belong to any of the 1000 classes of ImageNet-1K (ILSVRC2012), which makes NINCO suitable for evaluating out-of-distribution detection on ImageNet-1K . The NINCO main dataset consists of 64 OOD classes with a total of 5879 samples. These OOD classes were selected to have no categorical overlap with any classes of ImageNet-1K. Each sample was inspected individually by the authors to not contain ID objects. Besides NINCO, included are (in the same .tar.gz file) truly OOD versions of 11 popular OOD datasets with in total 2715 OOD samples. Further included are 17 OOD unit-tests, with 400 samples each. Code for loading and evaluating on each of the three datasets is provided at https://github.com/j-cb/NINCO. When using NINCO, please consider citing (besides the bibtex given below) the following data sources that were used to create NINCO: Hendrycks et al.: ”Scaling out-of-distribution detection for real-world settings”, ICML, 2022. Bossard et al.: ”Food-101 – mining discriminative components with random forests”, ECCV 2014. Zhou et al.: ”Places: A 10 million image database for scene recognition”, IEEE PAMI 2017. Huang et al.: ”Mos: Towards scaling out-of-distribution detection for large semantic space”, CVPR 2021. Li et al.: ”Caltech 101 (1.0)”, 2022. Ismail et al.: ”MYNursingHome: A fully-labelled image dataset for indoor object classification.”, Data in Brief (V. 32) 2020. The iNaturalist project: https://www.inaturalist.org/ When using NINCO_popular_datasets_subsamples, additionally to the above, please consider citing: Cimpoi et al.: ”Describing textures in the wild”, CVPR 2014. Hendrycks et al.: ”Natural adversarial examples”, CVPR 2021. Wang et al.: ”Vim: Out-of-distribution with virtual-logit matching”, CVPR 2022. Bendale et al.: ”Towards Open Set Deep Networks”, CVPR 2016. Vaze et al.: ”Open-set Recognition: a Good Closed-set Classifier is All You Need?”, ICLR 2022. Wang et al.: ”Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition.” ICML, 2022. Galil et al.: “A framework for benchmarking Class-out-of-distribution detection and its application to ImageNet”, ICLR 2023.

A large-scale curated dataset of over 152 million tweets, growing daily, related to COVID-19 chatter generated from January 1st to April 4th at the time of writing. Source: A large-scale COVID-19 Twitter chatter dataset for open scientific research -- an international collaboration

OpenImage-O is built for the ID dataset ImageNet-1k. It is manually annotated, comes with a naturally diverse distribution, and has a large scale. It is built to overcome several shortcomings of existing OOD benchmarks. OpenImage-O is image-by-image filtered from the test set of OpenImage-V3, which has been collected from Flickr without a predefined list of class names or tags, leading to natural class statistics and avoiding an initial design bias.

ADE-OoD is a public benchmark for dense out-of-distribution detection in general natural images. It measures the ability to detect and localize objects which are out-of-distribution with respect to the 150 categories of the ADE20k semantic segmentation dataset. The benchmark data and annotations are available for download at the project page: ade-ood.github.io/

The NINCO (No ImageNet Class Objects) dataset is introduced in the ICML 2023 paper In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation. The images in this dataset are free from objects that belong to any of the 1000 classes of ImageNet-1K (ILSVRC2012), which makes NINCO suitable for evaluating out-of-distribution detection on ImageNet-1K . The NINCO main dataset consists of 64 OOD classes with a total of 5879 samples. These OOD classes were selected to have no categorical overlap with any classes of ImageNet-1K. Each sample was inspected individually by the authors to not contain ID objects. Besides NINCO, included are (in the same .tar.gz file) truly OOD versions of 11 popular OOD datasets with in total 2715 OOD samples. Further included are 17 OOD unit-tests, with 400 samples each. Code for loading and evaluating on each of the three datasets is provided at https://github.com/j-cb/NINCO. When using NINCO, please consider citing (besides the bibtex given below) the following data sources that were used to create NINCO: Hendrycks et al.: ”Scaling out-of-distribution detection for real-world settings”, ICML, 2022. Bossard et al.: ”Food-101 – mining discriminative components with random forests”, ECCV 2014. Zhou et al.: ”Places: A 10 million image database for scene recognition”, IEEE PAMI 2017. Huang et al.: ”Mos: Towards scaling out-of-distribution detection for large semantic space”, CVPR 2021. Li et al.: ”Caltech 101 (1.0)”, 2022. Ismail et al.: ”MYNursingHome: A fully-labelled image dataset for indoor object classification.”, Data in Brief (V. 32) 2020. The iNaturalist project: https://www.inaturalist.org/ When using NINCO_popular_datasets_subsamples, additionally to the above, please consider citing: Cimpoi et al.: ”Describing textures in the wild”, CVPR 2014. Hendrycks et al.: ”Natural adversarial examples”, CVPR 2021. Wang et al.: ”Vim: Out-of-distribution with virtual-logit matching”, CVPR 2022. Bendale et al.: ”Towards Open Set Deep Networks”, CVPR 2016. Vaze et al.: ”Open-set Recognition: a Good Closed-set Classifier is All You Need?”, ICLR 2022. Wang et al.: ”Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition.” ICML, 2022. Galil et al.: “A framework for benchmarking Class-out-of-distribution detection and its application to ImageNet”, ICLR 2023.

Icons-50 is a dataset for studying surface variation robustness. Source: Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations

MultiOOD is the first benchmark for Multimodal OOD Detection and covers diverse dataset sizes and modalities. MultiOOD comprises five video datasets with over 85, 000 video clips in total. The datasets vary in the number of classes, ranging from 7 to 229, and in size, spanning from 3k to 57k. Video, optical flow, and audio are used as different types of modalities.

Pano3D is a new benchmark for depth estimation from spherical panoramas. Its goal is to drive progress for this task in a consistent and holistic manner. The Pano3D 360 depth estimation benchmark provides a standard Matterport3D train and test split, as well as a secondary GibsonV2 partioning for testing and training as well. The latter is used for zero-shot cross dataset transfer performance assessment and decomposes it into 3 different splits, each one focusing on a specific generalization axis.

This dataset was presented as part of the ICLR 2023 paper 𝘈 𝘧𝘳𝘢𝘮𝘦𝘸𝘰𝘳𝘬 𝘧𝘰𝘳 𝘣𝘦𝘯𝘤𝘩𝘮𝘢𝘳𝘬𝘪𝘯𝘨 𝘊𝘭𝘢𝘴𝘴-𝘰𝘶𝘵-𝘰𝘧-𝘥𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘪𝘰𝘯 𝘥𝘦𝘵𝘦𝘤𝘵𝘪𝘰𝘯 𝘢𝘯𝘥 𝘪𝘵𝘴 𝘢𝘱𝘱𝘭𝘪𝘤𝘢𝘵𝘪𝘰𝘯 𝘵𝘰 𝘐𝘮𝘢𝘨𝘦𝘕𝘦𝘵. It is a framework that, based on this dataset (a subset of the ImageNet-21k dataset) is able to generate a C-OOD (AKA open-set recognition) benchmark that covers a variety of difficulty levels. these benchmarks are tailored to the evaluated model. This approach provides a more accurate representation of the model’s own performance. The resulting difficulty levels of our framework allow benchmarking with respect to the difficulty levels most relevant to the task. For example, for a task with a high tolerance for risk (e.g., a task for an entertainment application), the performance of a model on a median difficulty level might be more important than on the hardest difficulty level (severity 10). The opposite might be true for some applications with a low tolerance for risk (e.g., medical applications), for which one requires the best performance to be attained even if the OOD is very hard to detect (severity 10). The paper in which the framework was introduced showed that detection algorithms do not always improve performance on all inputs equally, and could even hurt performance for specific difficulty levels and models. Choosing the combination of (model, detection algorithm) based only on the detection performance on all data may yield sub-optimal results for our specific desired level of difficulty.

A genomics dataset for OOD detection that allows other researchers to benchmark progress on this important problem. Source: Likelihood Ratios for Out-of-Distribution Detection

Simulated pulse Doppler radar signatures for four classes of helicopter-like targets. The classes differ in the number of rotating blades each kind of target carries, thus each class translates into a specific modulation pattern on the Doppler signature. Doppler signatures are a typical feature used to achieve radar targets discrimination. This dataset was generated using a simple open-source MATLAB simulation code, which can be easily modified to generate custom datasets with more classes and increased intra-class diversity. Dataset can be easily used for supervised classification, out-of-distribution detection (near and far), unsupervised learning and modulation pattern segmentation. The code includes the generation of an SPD representation for each signature, thanks to the computation of a covariance matrix, thus allowing for second-order specific data processing (e.g. Riemannian neural network or tangent PCA). Dataset used in the paper Dataset generation code