The ImageNet dataset contains 14,197,122 annotated images according to the WordNet hierarchy. Since 2010 the dataset is used in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), a benchmark in image classification and object detection. The publicly released dataset contains a set of manually annotated training images. A set of test images is also released, with the manual annotations withheld. ILSVRC annotations fall into one of two categories: (1) image-level annotation of a binary label for the presence or absence of an object class in the image, e.g., “there are cars in this image” but “there are no tigers,” and (2) object-level annotation of a tight bounding box and class label around an object instance in the image, e.g., “there is a screwdriver centered at position (20,25) with width of 50 pixels and height of 30 pixels”. The ImageNet project does not own the copyright of the images, therefore only thumbnails and URLs of images are provided. * Total number of non-empty WordNet synsets: 21841 * Total number of images: 14197122 * Number of images with bounding box annotations: 1,034,908 * Number of synsets with SIFT features: 1000 * Number of images with SIFT features: 1.2 million Source: ImageNet Large Scale Visual Recognition Challenge Image Source: https://cs.stanford.edu/people/karpathy/cnnembed/

The CIFAR-100 dataset (Canadian Institute for Advanced Research, 100 classes) is a subset of the Tiny Images dataset and consists of 60000 32x32 color images. The 100 classes in the CIFAR-100 are grouped into 20 superclasses. There are 600 images per class. Each image comes with a "fine" label (the class to which it belongs) and a "coarse" label (the superclass to which it belongs). There are 500 training images and 100 testing images per class. The criteria for deciding whether an image belongs to a class were as follows: * The class name should be high on the list of likely answers to the question “What is in this picture?” * The image should be photo-realistic. Labelers were instructed to reject line drawings. * The image should contain only one prominent instance of the object to which the class refers. * The object may be partially occluded or seen from an unusual viewpoint as long as its identity is still clear to the labeler. Source: https://www.cs.toronto.edu/~kriz/cifar.html Image Source: https://www.cs.toronto.edu/~kriz/cifar.html

ImageNet-C is an open source data set that consists of algorithmically generated corruptions (blur, noise) applied to the ImageNet test-set. Source: Selective Brain Damage: Measuring the Disparate Impact of Model Pruning Image Source: https://arxiv.org/pdf/1807.01697.pdf

The ImageNet-A dataset consists of real-world, unmodified, and naturally occurring examples that are misclassified by ResNet models. Source: On Robustness and Transferability of Convolutional Neural Networks Image Source: https://github.com/hendrycks/natural-adv-examples

The Stylized-ImageNet dataset is created by removing local texture cues in ImageNet while retaining global shape information on natural images via AdaIN style transfer. This nudges CNNs towards learning more about shapes and less about local textures. Source: Adversarial Examples Improve Image Recognition Image Source: https://github.com/rgeirhos/Stylized-ImageNet

Adversarial GLUE (AdvGLUE) is a new multi-task benchmark to quantitatively and thoroughly explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. In particular, we systematically apply 14 textual adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. Description from: Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models

ImageNet-Patch: A Dataset for Benchmarking Machine Learning Robustness against Adversarial Patches Adversarial patches are optimized contiguous pixel blocks in an input image that cause a machine-learning model to misclassify it. However, their optimization is computationally demanding, and requires careful hyperparameter tuning, potentially leading to suboptimal robustness evaluations. To overcome these issues, we propose ImageNet-Patch, a dataset to benchmark machine-learning models against adversarial patches. It consists of a set of patches, optimized to generalize across different models, and readily applicable to ImageNet data after preprocessing them with affine transformations. This process enables an approximate yet faster robustness evaluation, leveraging the transferability of adversarial perturbations.

CLEAR-Bias is a benchmark dataset designed to evaluate the robustness of large language models (LLMs) against bias elicitation, particularly under adversarial conditions. It comprises 4,400 prompts across two task formats: multiple-choice and sentence completion. These prompts span seven core bias categories—age, disability, ethnicity, gender, religion, sexual orientation, and socioeconomic status—as well as three intersectional categories, enabling the exploration of overlapping social biases often overlooked in standard evaluations. Each category includes 20 carefully crafted base prompts (10 per task type), which are further expanded using seven jailbreak techniques: machine translation, obfuscation, prefix and prompt injection, refusal suppression, reward incentives, and role-playing—each implemented with three variants. CLEAR-Bias is intended for researchers, developers, and practitioners seeking to assess or enhance the ethical behavior of language models. It serves as a benchmarking tool for measuring how effectively different models resist producing biased outputs in both standard and adversarial scenarios. By supporting the evaluation of ethical reliability before real-world deployment, CLEAR-Bias contributes to the development of safer and more responsible LLMs.

AdvSuffixes - Information AdvSuffixes is a curated dataset of adversarial prompts and suffixes designed to evaluate and enhance the robustness of large language models (LLMs) against adversarial attacks. By appending these suffixes to standard prompts, researchers and developers can explore and analyze how LLMs respond to potentially harmful input scenarios. This dataset is heavily inspired by AdvBench. ## Dataset Structure The dataset is organized as follows: data/ │ ├── advsuffixes/ │ ├── advsuffixes.csv # Adversarial suffixes and their respective prompts │ ├── advsuffixes_eval.txt # 100 additional evaluation prompts that are out-of-distribution │ ├── ... - advsuffixes.csv: The primary dataset containing pairs of adversarial suffixes and corresponding prompts. - advsuffixes_eval.txt: A set of 100 additional evaluation prompts designed to test model robustness, that are out-of-distribution from the original 519 prompts in advsuffixes.csv. Details about the dataset generation have been provided in Appendix B in the supplementary materials of the paper. There are 11763 listed suffixes overall, averaging 22.6 suffixes per prompt. --- ## License This dataset is distributed under the GNU General Public License v3.0.

This is the paper “DF-RAP: A Robust Adversarial Perturbation for Defending against Deepfakes in Real-world Social Network Scenarios" OSN-transmission CelebA sampling dataset collected by manual upload and download. This dataset includes 30,000 facial images of size 256×256 transmitted through online social networks (OSN) and their corresponding original images. Among them, Facebook, Twitter, WeChat and Weibo were selected as the transmission OSN, with 7500 images each.

SDoH Human Annotated Demoographic Robustness (SHADR) Dataset ## Overview The Social determinants of health (SDoH) play a pivotal role in determining patient outcomes. However, their documentation in electronic health records (EHR) remains incomplete. This dataset was created from a study examining the capability of large language models in extracting SDoH from the free text sections of EHRs. Furthermore, the study delved into the potential of synthetic clinical text to bolster the extraction process of these scarcely documented, yet crucial, clinical data. ## Dataset Structure & Modification To understand potential biases in high-performing models and in those pre-trained on general text, GPT-4 was utilized to infuse demographic descriptors into our synthetic data. For instance: - Original Sentence: "Widower admits fears surrounding potential judgment…" - Modified Sentence: "Hispanic widower admits fears surrounding potential judgment..." Such demographic-infused sentences underwent manual validation. Out of these: - 419 had mentions of SDoH - 253 had mentions of adverse SDoH - The remainder were tagged as NO_SDoH ## Instructions for Model Evaluation 1. Initially, run your model inference on the original sentences. 2. Subsequently, apply the same model to infer on the demographic-modified sentences. 3. Perform comparisons for robustness. For a detailed understanding of the "adverse" labeling, refer to https://arxiv.org/pdf/2308.06354.pdf. Here, the 'adverse' column demarcates if the label corresponds to an "adverse" or "non-adverse" SDoH. ## Current Performance Metrics - Best Model Performance: - Any SDoH: 88% Macro-F1 - Adverse SDoH: 84% Macro-F1 - Robustness Rate: - Any SDoH: 9.9% - Adverse SDoH: 14.3% --- How to Cite: @misc{guevara2023large, title={Large Language Models to Identify Social Determinants of Health in Electronic Health Records}, author={Marco Guevara and Shan Chen and Spencer Thomas and Tafadzwa L. Chaunzwa and Idalid Franco and Benjamin Kann and Shalini Moningi and Jack Qian and Madeleine Goldstein and Susan Harper and Hugo JWL Aerts and Guergana K. Savova and Raymond H. Mak and Danielle S. Bitterman}, year={2023}, eprint={2308.06354}, archivePrefix={arXiv}, primaryClass={cs.CL} }

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset