Active Learning

The COCO (Common Objects in Context) dataset is a large-scale object detection, segmentation, and captioning dataset. It is designed to encourage research on a wide variety of object categories and is commonly used for benchmarking computer vision models. It is an essential dataset for researchers and developers working on object detection, segmentation, and pose estimation tasks.

MNIST8M is derived from the MNIST dataset by applying random deformations and translations to the dataset. Source: Scalable and Sustainable Deep Learningvia Randomized Hashing

The DeepWeeds dataset consists of 17,509 images capturing eight different weed species native to Australia in situ with neighbouring flora. Source: https://github.com/AlexOlsen/DeepWeeds Image Source: https://github.com/AlexOlsen/DeepWeeds

The Fudan-ShanghaiTech dataset (FDST) is a dataset for video crowd counting. It contains 15K frames with about 394K annotated heads captured from 13 different scenes Source: https://arxiv.org/abs/1907.07911

DialoGLUE is a natural language understanding benchmark for task-oriented dialogue designed to encourage dialogue research in representation-based transfer, domain adaptation, and sample-efficient task learning. It consisting of 7 task-oriented dialogue datasets covering 4 distinct natural language understanding tasks. Source: DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue

The Groove MIDI Dataset (GMD) is composed of 13.6 hours of aligned MIDI and (synthesized) audio of human-performed, tempo-aligned expressive drumming. The dataset contains 1,150 MIDI files and over 22,000 measures of drumming.

A benchmark which bridges the gap between freely available, documented, and motivated artificial benchmarks and properties of real industrial problems. The resulting industrial benchmark (IB) has been made publicly available to the RL community by publishing its Java and Python code, including an OpenAI Gym wrapper, on Github. Source: A Benchmark Environment Motivated by Industrial Control Problems

HJDataset is a large dataset of Historical Japanese Documents with Complex Layouts. It contains over 250,000 layout element annotations of seven types. In addition to bounding boxes and masks of the content regions, it also includes the hierarchical structures and reading orders for layout elements. The dataset is constructed using a combination of human and machine efforts. Source: A Large Dataset of Historical Japanese Documents with Complex Layouts Image Source: https://dell-research-harvard.github.io/HJDataset/

A dataset for evaluating text classification, domain adaptation, and active learning models. The dataset consists of 22,660 documents (tweets) collected in 2018 and 2019. It spans across four domains: Alzheimer's, Parkinson's, Cancer, and Diabetes.

Specially designed to evaluate active learning for video object detection in road scenes. Source: Temporal Coherence for Active Learning in Videos

COMP6 is a benchmark for evaluating the extensibility of machine-learning based molecular potentials. It contains a diverse set of organic molecules. Source: https://github.com/isayev/COMP6

Arxiv GR-QC (General Relativity and Quantum Cosmology) collaboration network is from the e-print arXiv and covers scientific collaborations between authors papers submitted to General Relativity and Quantum Cosmology category. If an author i co-authored a paper with author j, the graph contains a undirected edge from i to j. If the paper is co-authored by k authors this generates a completely connected (sub)graph on k nodes. Source: https://snap.stanford.edu/data/ca-GrQc.html

Goldfinch is a dataset for fine-grained recognition challenges. It contains a list of bird, butterfly, aircraft, and dog categories with relevant Google image search and Flickr search URLs. In addition, it also includes a set of active learning annotations on dog categories. Source: The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition

A benchmark for molecular machine learning where improvements in model performance can be immediately observed in the throughput of promising molecules synthesized in the lab. Photoswitches are a versatile class of molecule for medical and renewable energy applications where a molecule's efficacy is governed by its electronic transition wavelengths. Source: The Photoswitch Dataset: A Molecular Machine Learning Benchmark for the Advancement of Synthetic Chemistry

The L-Bird (Large-Bird) dataset contains nearly 4.8 million images which are obtained by searching images of a total of 10,982 bird species from the Internet. Source: Fine-Grained Visual Categorization using Meta-Learning Optimization with Sample Selection of Auxiliary Data Image Source: https://arxiv.org/pdf/1511.06789.pdf

Annotating data is a time-consuming and costly task, but it is inherently required for supervised machine learning. Active Learning (AL) is an established method that minimizes human labeling effort by iteratively selecting the most informative unlabeled samples for expert annotation, thereby improving the overall classification performance. Even though AL has been known for decades [1], AL is still rarely used in real-world applications. As indicated in the two community web surveys among the NLP community about AL [2], [3], two main reasons continue to hold practitioners back from using AL: first, the complexity of setting AL up, and second, a lack of trust in its effectiveness. We hypothesize that both reasons share the same culprit: the large hyperparameter space of AL. This mostly unexplored hyperparameter space often leads to misleading and irreproducible AL experiment results. In this study, we first compiled a large hyperparameter grid of over 4.6 million hyperparameter combinations, second, recorded the performance of all combinations in the so-far biggest conducted AL study, and third, analyzed the impact of each hyperparameter in the experiment results. Besides our publication we are making our raw experiment results publically available for other researchers to built upon. [1] B. Settles, “Active Learning Literature Survey,” University of Wisconsin-Madison Department of Computer Sciences, Technical Report, 2009. [2] K. Tomanek and F. Olsson, “A web survey on the use of active learning to support annotation of text data,” in Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, E. Ringger, R. Haertel, and K. Tomanek, Eds. Boulder, Colorado: Association for Computational Linguistics, 2009, pp. 45–48. [3] J. Romberg, C. Schröder, J. Gonsior, K. Tomanek, and F. Olsson, “Have LLMs Made Active Learning Obsolete? Surveying the NLP Community,” 2025.