Continual Learning

The CIFAR-100 dataset (Canadian Institute for Advanced Research, 100 classes) is a subset of the Tiny Images dataset and consists of 60000 32x32 color images. The 100 classes in the CIFAR-100 are grouped into 20 superclasses. There are 600 images per class. Each image comes with a "fine" label (the class to which it belongs) and a "coarse" label (the superclass to which it belongs). There are 500 training images and 100 testing images per class. The criteria for deciding whether an image belongs to a class were as follows: * The class name should be high on the list of likely answers to the question “What is in this picture?” * The image should be photo-realistic. Labelers were instructed to reject line drawings. * The image should contain only one prominent instance of the object to which the class refers. * The object may be partially occluded or seen from an unusual viewpoint as long as its identity is still clear to the labeler. Source: https://www.cs.toronto.edu/~kriz/cifar.html Image Source: https://www.cs.toronto.edu/~kriz/cifar.html

The MNIST database (Modified National Institute of Standards and Technology database) is a large collection of handwritten digits. It has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger NIST Special Database 3 (digits written by employees of the United States Census Bureau) and Special Database 1 (digits written by high school students) which contain monochrome images of handwritten digits. The digits have been size-normalized and centered in a fixed-size image. The original black and white (bilevel) images from NIST were size normalized to fit in a 20x20 pixel box while preserving their aspect ratio. The resulting images contain grey levels as a result of the anti-aliasing technique used by the normalization algorithm. the images were centered in a 28x28 image by computing the center of mass of the pixels, and translating the image so as to position this point at the center of the 28x28 field. Source: http://yann.lecun.com/exdb/mnist/ Image Source: https://en.wikipedia.org/wiki/MNIST_database#/media/File:MnistExamples.png

Cityscapes is a large-scale database which focuses on semantic understanding of urban street scenes. It provides semantic, instance-wise, and dense pixel annotations for 30 classes grouped into 8 categories (flat surfaces, humans, vehicles, constructions, objects, nature, sky, and void). The dataset consists of around 5000 fine annotated images and 20000 coarse annotated ones. Data was captured in 50 cities during several months, daytimes, and good weather conditions. It was originally recorded as video so the frames were manually selected to have the following features: large number of dynamic objects, varying scene layout, and varying background. Source: A Review on Deep Learning Techniques Applied to Semantic Segmentation Image Source: https://www.cityscapes-dataset.com/dataset-overview/

The Caltech-UCSD Birds-200-2011 (CUB-200-2011) dataset is the most widely-used dataset for fine-grained visual categorization task. It contains 11,788 images of 200 subcategories belonging to birds, 5,994 for training and 5,794 for testing. Each image has detailed annotations: 1 subcategory label, 15 part locations, 312 binary attributes and 1 bounding box. The textual information comes from Reed et al.. They expand the CUB-200-2011 dataset by collecting fine-grained natural language descriptions. Ten single-sentence descriptions are collected for each image. The natural language descriptions are collected through the Amazon Mechanical Turk (AMT) platform, and are required at least 10 words, without any information of subcategories and actions. Source: Fine-grained Visual-textual Representation Learning Image Source: http://www.vision.caltech.edu/visipedia/CUB-200-2011.html

mini-Imagenet is proposed by Matching Networks for One Shot Learning . In NeurIPS, 2016. This dataset consists of 50000 training images and 10000 testing images, evenly distributed across 100 classes.

Oxford 102 Flower is an image classification dataset consisting of 102 flower categories. The flowers chosen to be flower commonly occurring in the United Kingdom. Each class consists of between 40 and 258 images. The images have large scale, pose and light variations. In addition, there are categories that have large variations within the category and several very similar categories.

The ADE20K semantic segmentation dataset contains more than 20K scene-centric images exhaustively annotated with pixel-level objects and object parts labels. There are totally 150 semantic categories, which include stuffs like sky, road, grass, and discrete objects like person, car, bed. Source: Cooperative Image Segmentation and Restoration in Adverse Environmental Conditions Image Source: https://groups.csail.mit.edu/vision/datasets/ADE20K/

The Stanford Cars dataset consists of 196 classes of cars with a total of 16,185 images, taken from the rear. The data is divided into almost a 50-50 train/test split with 8,144 training images and 8,041 testing images. Categories are typically at the level of Make, Model, Year. The images are 360×240. Source: View Independent Vehicle Make, Model and Color Recognition Using Convolutional Neural Network Image Source: https://ai.stanford.edu/~jkrause/cars/car_dataset.html

CoNLL-2003 is a named entity recognition dataset released as a part of CoNLL-2003 shared task: language-independent named entity recognition. The data consists of eight files covering two languages: English and German. For each of the languages there is a training file, a development file, a test file and a large file with unannotated data. The English data was taken from the Reuters Corpus. This corpus consists of Reuters news stories between August 1996 and August 1997. For the training and development set, ten days worth of data were taken from the files representing the end of August 1996. For the test set, the texts were from December 1996. The preprocessed raw data covers the month of September 1996. The text for the German data was taken from the ECI Multilingual Text Corpus. This corpus consists of texts in many languages. The portion of data that was used for this task, was extracted from the German newspaper Frankfurter Rundshau. All three of the training, development and test sets were taken from articles written in one week at the end of August 1992. The raw data were taken from the months of September to December 1992. | English data | Articles | Sentences | Tokens | LOC | MISC | ORG | PER | |-------------------|----------|-----------|---------|------|------|------|------| | Training set | 946 | 14,987 | 203,621 | 7140 | 3438 | 6321 | 6600 | | Development set | 216 | 3,466 | 51,362 | 1837 | 922 | 1341 | 1842 | | Test set | 231 | 3,684 | 46,435 | 1668 | 702 | 1661 | 1617 | Number of articles, sentences, tokens and entities (locations, miscellaneous, organizations, and persons) in English data files. | German data | Articles | Sentences | Tokens | LOC | MISC | ORG | PER | |-------------------|----------|-----------|---------|------|------|------|------| | Training set | 553 | 12,705 | 206,931 | 4363 | 2288 | 2427 | 2773 | | Development set | 201 | 3,068 | 51,444 | 1181 | 1010 | 1241 | 1401 | | Test set | 155 | 3,160 | 51,943 | 1035 | 670 | 773 | 1195 | Number of articles, sentences, tokens and entities (locations, miscellaneous, organizations, and persons) in German data files.

OntoNotes 5.0 is a large corpus comprising various genres of text (news, conversational telephone speech, weblogs, usenet newsgroups, broadcast, talk shows) in three languages (English, Chinese, and Arabic) with structural information (syntax and predicate argument structure) and shallow semantics (word sense linked to an ontology and coreference). OntoNotes Release 5.0 contains the content of earlier releases - and adds source data from and/or additional annotations for, newswire, broadcast news, broadcast conversation, telephone conversation and web data in English and Chinese and newswire data in Arabic. Source: https://catalog.ldc.upenn.edu/LDC2013T19

The Sketch dataset contains over 20,000 sketches evenly distributed over 250 object categories. Source: http://cybertron.cg.tu-berlin.de/eitz/projects/classifysketch/ Image Source: http://cybertron.cg.tu-berlin.de/eitz/projects/classifysketch/

The CoNLL dataset is a widely used resource in the field of natural language processing (NLP). The term “CoNLL” stands for Conference on Natural Language Learning. It originates from a series of shared tasks organized at the Conferences of Natural Language Learning.

CORe50 is a dataset designed for assessing Continual Learning techniques in an Object Recognition context. Source: CORe50: a New Dataset and Benchmark for Continuous Object Recognition

Permuted MNIST is an MNIST variant that consists of 70,000 images of handwritten digits from 0 to 9, where 60,000 images are used for training, and 10,000 images for test. The difference of this dataset from the original MNIST is that each of the ten tasks is the multi-class classification of a different random permutation of the input pixels. Source: Lifelong Learning with Dynamically Expandable Networks Image Source: https://arxiv.org/pdf/1810.12488.pdf

WikiArt contains painting from 195 different artists. The dataset has 42129 images for training and 10628 images for testing. Source: Adding New Tasks to a Single Network with Weight Transformations using Binary Masks Image Source: https://towardsdatascience.com/the-non-treachery-of-dataset-df1f6cbe577e

AIDS is a graph dataset. It consists of 2000 graphs representing molecular compounds which are constructed from the AIDS Antiviral Screen Database of Active Compounds. It contains 4395 chemical compounds, of which 423 belong to class CA, 1081 to CM, and the remaining compounds to CI. Source: DGCNN: Disordered Graph Convolutional Neural Network Based on the Gaussian Mixture Model Image Source: https://www.researchgate.net/figure/Sample-component-in-AIDS-kernel-dataset-with-Graphwave-based-structural-role-colors-Here_fig1_338282222

ROAD is designed to test an autonomous vehicle's ability to detect road events, defined as triplets composed by an active agent, the action(s) it performs and the corresponding scene locations. ROAD comprises videos originally from the Oxford RobotCar Dataset, annotated with bounding boxes showing the location in the image plane of each road event.

With social media becoming increasingly popular on which lots of news and real-time events are reported, developing automated question answering systems is critical to the effectiveness of many applications that rely on real-time knowledge. While previous question answering (QA) datasets have concentrated on formal text like news and Wikipedia, the first large-scale dataset for QA over social media data is presented. To make sure the tweets are meaningful and contain interesting information, tweets used by journalists to write news articles are gathered. Then human annotators are asked to write questions and answers upon these tweets. Unlike other QA datasets like SQuAD in which the answers are extractive, the answer are allowed to be abstractive. The task requires model to read a short tweet and a question and outputs a text phrase (does not need to be in the tweet) as the answer. Source: TWEETQA: A Social Media Focused Question Answering Dataset

2010 i2b2/VA is a biomedical dataset for relation classification and entity typing.

Continual World is a benchmark consisting of realistic and meaningfully diverse robotic tasks built on top of Meta-World as a testbed.

This dataset has 20 classes and each class has about 1000 documents. The data split for train/validation/test is 1600/200/200. We created 10 tasks, 2 classes per task. Since this is topic-based text classification data, the classes are very different and have little shared knowledge. As mentioned above, this application (and dataset) is mainly used to show a CL model's ability to overcome forgetting. Detailed statistics please on page https://github.com/ZixuanKe/PyContinual

A set of 19 ASC datasets (reviews of 19 products) producing a sequence of 19 tasks. Each dataset represents a task. The datasets are from 4 sources: (1) HL5Domains (Hu and Liu, 2004) with reviews of 5 products; (2) Liu3Domains (Liu et al., 2015) with reviews of 3 products; (3) Ding9Domains (Ding et al., 2008) with reviews of 9 products; and (4) SemEval14 with reviews of 2 products - SemEval 2014 Task 4 for laptop and restaurant. For (1), (2) and (3), we split about 10% of the original data as the validate data, another about 10% of the original data as the testing data. For (4), We use 150 examples from the training set for validation. To be consistent with existing research(Tang et al., 2016), examples belonging to the conflicting polarity (both positive and negative sentiments are expressed about an aspect term) are not used. Statistics and details of the 19 datasets are given on Page https://github.com/ZixuanKe/PyContinual.

LReID is a benchmark for lifelong person reidentification. It has been built using existing datasets, and it consists of two subsets: LReID-Seen and LReID-Unseen. LReID-Seen contains 40,459 training images of the 2,500 identities selected from the following datasets: CUHK03, Market-1501, MSMT17 V2, DukeMTMC-ReID, CUHK-SYSU ReID. This is used to test a model's performance on seen domains. LReID-Unseen contains 9,854 images from 3,594 identities from the following datasets: VIPeR, PRID, GRID, i-LIDS, CUHK01, CUHK02, SenseReID. This subset is used to test the model's generalisation capabilities to unseen domains.

The goal of this challenge is to solve simultaneously ten image classification problems representative of very different visual domains. The data for each domain is obtained from the following image classification benchmarks: ImageNet CIFAR-100 Aircraft Daimler pedestrian classification Describable textures German traffic signs Omniglot SVHN UCF101 Dynamic Images VGG-Flowers The union of the images from the ten datasets is split in training, validation, and test subsets. Different domains contain different image categories as well as a different number of images. The task is to train the best possible classifier to address all ten classification tasks using the training and validation subsets, apply the classifier to the test set, and send us the resulting annotation file for assessment. The winner will be determined based on a weighted average of the classification performance on each domain, using the scoring scheme described below. At test time, your model is allowed to know the ground-truth domain of each test image (ImageNet, CIFAR-100, ...) but, of course, not its category.

A set of 10 DSC datasets (reviews of 10 products) to produce sequences of tasks. The products are Sports, Toys, Tools, Video, Pet, Musical, Movies, Garden, Offices, and Kindle. 2500 positive and 2500 negative training reviews per task . The validation reviews are with 250 positive and 250 negative and the test reviews are with 250 positive and 250 negative reviews. The detailed statistic on page https://github.com/ZixuanKe/PyContinual

F-CelebA - This dataset is adapted from federated learning. Federated learning is an emerging machine learning paradigm with an emphasis on data privacy. The idea is to train through model aggregation rather than conventional data aggregation and keep local data staying on the local device. This dataset naturally consists of similar tasks and each of the 10 tasks contains images of a celebrity labeled by whether he/she is smiling or not. More detailed please check page https://github.com/ZixuanKe/CAT

Wild-Time is a benchmark of 5 datasets that reflect temporal distribution shifts arising in a variety of real-world applications, including patient prognosis and news classification. On these datasets, we systematically benchmark 13 prior approaches, including methods in domain generalization, continual learning, self-supervised learning, and ensemble learning. Source: Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time Image Source: https://arxiv.org/pdf/2211.14238v1.pdf

TemporalWiki is a lifelong benchmark for ever-evolving LMs that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation, respectively. The benchmark hence allows researchers to periodically track an LM's ability to retain previous knowledge and acquire updated/new knowledge at each point in time.

HASY is a dataset of single symbols similar to MNIST. It contains 168,233 instances of 369 classes. HASY contains two challenges: A classification challenge with 10 pre-defined folds for 10-fold cross-validation and a verification challenge. Source: The HASYv2 dataset

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

UESTC-MMEA-CL is a new multi-modal activity dataset for continual egocentric activity recognition, which is proposed to promote future studies on continual learning for first-person activity recognition in wearable applications. Our dataset provides not only vision data with auxiliary inertial sensor data but also comprehensive and complex daily activity categories for the purpose of continual learning research. UESTC-MMEA-CL comprises 30.4 hours of fully synchronized first-person video clips, acceleration stream and gyroscope data in total. There are 32 activity classes in the dataset and each class contains approximately 200 samples. We divide the samples of each class into the training set, validation set and test set according to the ratio of 7:2:1. For the continual learning evaluation, we present three settings of incremental steps, i.e., the 32 classes are divided into {16, 8, 4} incremental steps and each step contains {2, 4, 8} activity classes, respectively.

The Industrial Objects in Varied Contexts (InVar) Dataset was internally produced by our team and contains 100 objects in 20800 total images (208 images per class). The objects consist of common automotive, machine and robotics lab parts. Each class contains 4 sub-categories (52 images each) with different attributes and visual complexities. White background (D_wh): The object is against a clean white background and the object is clear, centred and in focus. Stationary Setup (D_st): These images are also taken against a clean background using a stationary camera setup, with uncentered objects at a constant distance. The images have lower DPI resolution with occasional cropping. Handheld (D_ha): These images are taken with the user holding the objects, with occasional occluding. Cluttered background (D_cl): These images are taken with the object placed along with other objects from the lab in the background and no occlusion.

(L)ifel(O)ng (R)obotic V(IS)ion (OpenLORIS) - Object Recognition Dataset (OpenLORIS-Object) is designed for accelerating the lifelong/continual/incremental learning research and application，currently focusing on improving the continuous learning capability of the common objects in the home scenario. Source: OpenLORIS-object

SKILL-102 consists of 102 image classification datasets. Each one supports one complex classification task, and the corresponding dataset was obtained from previously published sources (e.g., task 1: classify flowers into 102 classes, such as lily, rose, petunia, etc using 8,185 train/val/test images (Nilsback & Zisserman, 2008a); task 2: classify 67 types of scenes, such as kitchen, bedroom, gas station, library, etc using 15,524 images (Quattoni & Torralba, 2009). In total, SKILL-102 comprises 102 tasks, 5,033 classes, and 2,041,225 training images. To the best of our knowledge, SKILL-102 is the most challenging completely real (not synthesized or permuted) image classification benchmark for LL and SKILL algorithms, with the largest number of tasks, number of classes, and inter-task variance.

BeGin provides 23 benchmark scenarios for graph from 14 real-world datasets, which cover 12 combinations of the incremental settings and the levels of problem. In addition, BeGin provides various basic evaluation metrics for measuring the performances and final evalution metrics designed for continual learning. Source: BeGin: Extensive Benchmark Scenarios and An Easy-to-use Framework for Graph Continual Learning Image Source: https://arxiv.org/pdf/2211.14568v1.pdf