The CIFAR-100 dataset (Canadian Institute for Advanced Research, 100 classes) is a subset of the Tiny Images dataset and consists of 60000 32x32 color images. The 100 classes in the CIFAR-100 are grouped into 20 superclasses. There are 600 images per class. Each image comes with a "fine" label (the class to which it belongs) and a "coarse" label (the superclass to which it belongs). There are 500 training images and 100 testing images per class. The criteria for deciding whether an image belongs to a class were as follows: * The class name should be high on the list of likely answers to the question “What is in this picture?” * The image should be photo-realistic. Labelers were instructed to reject line drawings. * The image should contain only one prominent instance of the object to which the class refers. * The object may be partially occluded or seen from an unusual viewpoint as long as its identity is still clear to the labeler. Source: https://www.cs.toronto.edu/~kriz/cifar.html Image Source: https://www.cs.toronto.edu/~kriz/cifar.html

WSJ0-2mix is a speech recognition corpus of speech mixtures using utterances from the Wall Street Journal (WSJ0) corpus. Source: Deep clustering: Discriminative embeddings for segmentation and separation

ImageNet-P consists of noise, blur, weather, and digital distortions. The dataset has validation perturbations; has difficulty levels; has CIFAR-10, Tiny ImageNet, ImageNet 64 × 64, standard, and Inception-sized editions; and has been designed for benchmarking not training networks. ImageNet-P departs from ImageNet-C by having perturbation sequences generated from each ImageNet validation image. Each sequence contains more than 30 frames, so to counteract an increase in dataset size and evaluation time only 10 common perturbations are used. Source: Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

NAS-Bench-1Shot1 draws on the recent large-scale tabular benchmark NAS-Bench-101 for cheap anytime evaluations of one-shot NAS methods. Source: NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search

comma 2k19 is a dataset of over 33 hours of commute in California's 280 highway. This means 2019 segments, 1 minute long each, on a 20km section of highway driving between California's San Jose and San Francisco. The dataset was collected using comma EONs that have sensors similar to those of any modern smartphone including a road-facing camera, phone GPS, thermometers and a 9-axis IMU. Source: A Commute in Data: The comma2k19 Dataset

Text Classification Attack Benchmark (TCAB) is a dataset for analyzing, understanding, detecting, and labeling adversarial attacks against text classifiers. TCAB includes 1.5 million attack instances, generated by twelve adversarial attack targeting three classifiers trained on six source datasets for sentiment analysis and abuse detection in English. The process of generating attacks is automated, so that TCAB can easily be extended to incorporate new text attacks and better classifiers as they are developed. Source: TCAB: A Large-Scale Text Classification Attack Benchmark

AdvSuffixes - Information AdvSuffixes is a curated dataset of adversarial prompts and suffixes designed to evaluate and enhance the robustness of large language models (LLMs) against adversarial attacks. By appending these suffixes to standard prompts, researchers and developers can explore and analyze how LLMs respond to potentially harmful input scenarios. This dataset is heavily inspired by AdvBench. ## Dataset Structure The dataset is organized as follows: data/ │ ├── advsuffixes/ │ ├── advsuffixes.csv # Adversarial suffixes and their respective prompts │ ├── advsuffixes_eval.txt # 100 additional evaluation prompts that are out-of-distribution │ ├── ... - advsuffixes.csv: The primary dataset containing pairs of adversarial suffixes and corresponding prompts. - advsuffixes_eval.txt: A set of 100 additional evaluation prompts designed to test model robustness, that are out-of-distribution from the original 519 prompts in advsuffixes.csv. Details about the dataset generation have been provided in Appendix B in the supplementary materials of the paper. There are 11763 listed suffixes overall, averaging 22.6 suffixes per prompt. --- ## License This dataset is distributed under the GNU General Public License v3.0.

The Cifar10Mnist dataset is created using CIFAR-10 and MNIST data sources. Since the CIFAR-10 training set consists of 50000 images and the MNIST training set contains 60000 digits, the first 50000 digits from MNIST are padded on top of the CIFAR-10 images after making them slightly translucent. A first training dataset is then obtained (50000 images). Furthermore, the remaining 10000 MNIST digits are padded on top of 10000 random CIFAR10 images (with a fixed seed). This gives the possibility of having a second training dataset of 60000 images. For the test set, the 10000 CIFAR-10 images are padded over the 10000 MNIST digits.

This dataset consists of 225 malicious tasks, which were integrated into ten distinct jailbreaking prompts. The malicious tasks were divided into five categories, namely, 1. Misinformation and Disinformation 2. Security Threats and Cybercrimes 3. Unlawful Behaviors and Activities 4. Hate Speech and Discrimination 5. Substance Abuse and Dangerous Practices. The jailbreaking prompts were carefully selected to cover a diverse range of scenarios. These scenarios included role-playing, simulations, attention-shifting, and privileged execution, and the placement of the malicious task within the jailbreaking prompts was also varied. List of malicious tasks only: https://github.com/CrystalEye42/eval-safety/blob/main/malicious_tasks_dataset.yaml Malicious tasks with jailbreaking prompts: https://github.com/CrystalEye42/eval-safety/blob/main/integrated.yaml

The PointDenoisingBenchmark dataset features 28 different shapes, split into 18 training shapes and 10 test shapes. * PointDenoisingBenchmark for outliers removal: contains noisy point clouds with different levels of gaussian noise and the corresponding clean ground truths. * PointDenoisingBenchmark for denoising: contains noisy point clouds with different levels of noise and density of outliers and the corresponding clean ground truths. Source: PointCleanNet: Learning to Denoise and Remove Outliers from Dense Point Clouds

REAP is a digital benchmark that allows the user to evaluate patch attacks on real images, and under real-world conditions. Built on top of the Mapillary Vistas dataset, the benchmark contains over 14,000 traffic signs. Each sign is augmented with a pair of geometric and lighting transformations, which can be used to apply a digitally generated patch realistically onto the sign. Source: REAP: A Large-Scale Realistic Adversarial Patch Benchmark Image Source: https://arxiv.org/pdf/2212.05680v1.pdf

SUDO is a benchmark of 50 real-world malicious tasks designed to evaluate LLM-based computer agents in live desktop and web environments. It covers critical risk domains—including system security, content safety, societal harms, and privacy violations—based on the AirBench taxonomy. The dataset supports fine-grained evaluation using task-specific checklists and can be used to assess model misuse potential, build safer agents, or guide alignment research.

Texygen is a benchmarking platform to support research on open-domain text generation models. Texygen has not only implemented a majority of text generation models, but also covered a set of metrics that evaluate the diversity, the quality and the consistency of the generated texts. The Texygen platform could help standardize the research on text generation and facilitate the sharing of fine-tuned open-source implementations among researchers for their work. As a consequence, this would help in improving the reproductivity and reliability of future research work in text generation. Source: Texygen Platform