Out of Distribution (OOD) Detection

The ATIS (Airline Travel Information Systems) is a dataset consisting of audio recordings and corresponding manual transcripts about humans asking for flight information on automated airline travel inquiry systems. The data consists of 17 unique intent categories. The original split contains 4478, 500 and 893 intent-labeled reference utterances in train, development and test set respectively. Source: Spoken Language Intent Detection using Confusion2Vec

The SNIPS Natural Language Understanding benchmark is a dataset of over 16,000 crowdsourced queries distributed among 7 user intents of various complexity: * SearchCreativeWork (e.g. Find me the I, Robot television show), * GetWeather (e.g. Is it windy in Boston, MA right now?), * BookRestaurant (e.g. I want to book a highly rated restaurant in Paris tomorrow night), * PlayMusic (e.g. Play the last track from Beyoncé off Spotify), * AddToPlaylist (e.g. Add Diamonds to my roadtrip playlist), * RateBook (e.g. Give 6 stars to Of Mice and Men), * SearchScreeningEvent (e.g. Check the showtimes for Wonder Woman in Paris). The training set contains of 13,084 utterances, the validation set and the test set contain 700 utterances each, with 100 queries per intent. Source: https://paperswithcode.com/paper/snips-voice-platform-an-embedded-spoken/

ImageNet-O consists of images from classes that are not found in the ImageNet-1k dataset. It is used to test the robustness of vision models to out-of-distribution samples. It's reported using the AUPR metric.

It is manually annotated, comes with a naturally diverse distribution, and has a large scale. It is built to overcome several shortcomings of existing OOD benchmarks. OpenImage-O is image-by-image filtered from the test set of OpenImage-V3, which has been collected from Flickr without a predefined list of class names or tags, leading to natural class statistics and avoiding an initial design bias.

The NINCO (No ImageNet Class Objects) dataset is introduced in the ICML 2023 paper In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation. The images in this dataset are free from objects that belong to any of the 1000 classes of ImageNet-1K (ILSVRC2012), which makes NINCO suitable for evaluating out-of-distribution detection on ImageNet-1K . The NINCO main dataset consists of 64 OOD classes with a total of 5879 samples. These OOD classes were selected to have no categorical overlap with any classes of ImageNet-1K. Each sample was inspected individually by the authors to not contain ID objects. Besides NINCO, included are (in the same .tar.gz file) truly OOD versions of 11 popular OOD datasets with in total 2715 OOD samples. Further included are 17 OOD unit-tests, with 400 samples each. Code for loading and evaluating on each of the three datasets is provided at https://github.com/j-cb/NINCO. When using NINCO, please consider citing (besides the bibtex given below) the following data sources that were used to create NINCO: Hendrycks et al.: ”Scaling out-of-distribution detection for real-world settings”, ICML, 2022. Bossard et al.: ”Food-101 – mining discriminative components with random forests”, ECCV 2014. Zhou et al.: ”Places: A 10 million image database for scene recognition”, IEEE PAMI 2017. Huang et al.: ”Mos: Towards scaling out-of-distribution detection for large semantic space”, CVPR 2021. Li et al.: ”Caltech 101 (1.0)”, 2022. Ismail et al.: ”MYNursingHome: A fully-labelled image dataset for indoor object classification.”, Data in Brief (V. 32) 2020. The iNaturalist project: https://www.inaturalist.org/ When using NINCO_popular_datasets_subsamples, additionally to the above, please consider citing: Cimpoi et al.: ”Describing textures in the wild”, CVPR 2014. Hendrycks et al.: ”Natural adversarial examples”, CVPR 2021. Wang et al.: ”Vim: Out-of-distribution with virtual-logit matching”, CVPR 2022. Bendale et al.: ”Towards Open Set Deep Networks”, CVPR 2016. Vaze et al.: ”Open-set Recognition: a Good Closed-set Classifier is All You Need?”, ICLR 2022. Wang et al.: ”Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition.” ICML, 2022. Galil et al.: “A framework for benchmarking Class-out-of-distribution detection and its application to ImageNet”, ICLR 2023.

The MUAD dataset (Multiple Uncertainties for Autonomous Driving), consisting of 10,413 realistic synthetic images with diverse adverse weather conditions (night, fog, rain, snow), out-of-distribution objects, and annotations for semantic segmentation, depth estimation, object, and instance detection. Predictive uncertainty estimation is essential for the safe deployment of Deep Neural Networks in real-world autonomous systems and MUAD allows to a better assess the impact of different sources of uncertainty on model performance.

The PATIS is a Persian language dataset for intent detection and slot filling.

The FathomNet2023 competition dataset is a subset of the broader FathomNet marine image repository. The training and test images for the competition were all collected in the Monterey Bay Area between the surface and 1300 meters depth by the Monterey Bay Aquarium Research Institute. The images contain bounding box annotations of 290 categories of bottom dwelling animals. The training and validation data are split across an 800 meter depth threshold: all training data is collected from 0-800 meters, evaluation data comes from the whole 0-1300 meter range. Since an organisms' habitat range is partially a function of depth, the species distributions in the two regions are overlapping but not identical. Test images are drawn from the same region but may come from above or below the depth horizon. The competition goal is to label the animals present in a given image (i.e. multi-label classification) and determine whether the image is out-of-sample.

In this dataset, various objects are arranged on a white table. A UR5e robot picks and place a target object specified on the title of the video/image sequence. Videos under auto- folder are collected with automatic operation of the robot. Videos under human- folders are collected with the tele-operation of the robot. Ground-truth tracking bounding boxes are generated with STARK, and when the target exits the camera frame, the bounding box estimation is switched to [-1, -1, -1, -1], indicating target not shown.