Systematic Generalization

SCAN is a dataset for grounded navigation which consists of a set of simple compositional navigation commands paired with the corresponding action sequences. Source: Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks

Grounded SCAN poses a simple task, where an agent must execute action sequences based on a synthetic language instruction. The agent is presented with a simple grid world containing a collection of objects, each of which is associated with a vector of features. The agent is evaluated on its ability to follow one or more instructions in this environment. Some instructions require interaction with particular kinds of objects. gSCAN Example

This dataset code generates mathematical question and answer pairs, from a range of question types at roughly school-level difficulty. This is designed to test the mathematical learning and algebraic reasoning skills of learning models. Source: DeepMind

The Probabilistic Context Free Grammar String Edit Task (PCFG SET) dataset is a dataset with sequence to sequence problems specifically designed to test different aspects of compositional generalisation. In particular, the dataset contains splits to test for systematicity, productivity, substitutivity, localism and overgeneralisation. The input alphabet of PCFG SET contains three types of words: words for unary and binary functions that represent \emph{string edit operations} (e.g. $\texttt{append}, \texttt{copy}, \texttt{reverse})$, elements to form the string sequences that these functions can be applied to (e.g. $\texttt{A}, \texttt{B}, \texttt{A1}, \texttt{B1}$), and a separator to separate the arguments of a binary function ($\texttt{,}$). The input sequences that are formed with this alphabet are sequences describing how a series of such operations are to be applied to a string argument. For instance: - $\texttt{repeat A B C }$ - $\texttt{echo remove\_first D K , E F}$ - $\texttt{append swap F G H , repeat I J}$ The input sequences are generated with a PCFG, whose production probabilities are learned with EM to match the depth and length distributions in a corpus with English sentences. The output of a PCFG SET sequence, representing its meaning, is constructed by recursively applying the string edit operations specified in the sequence. For instance: - $\texttt{repeat A B C }$ & $\rightarrow$ & $\texttt{A B C A B C}$ - $\texttt{echo remove\_first D K , E F}$ & $\rightarrow$ & $\texttt{E F F}$ - $\texttt{append swap F G H , repeat I J}$ & $\rightarrow$ & $\texttt{H G F I J I J }$ The string alphabet used for the construction of the dataset has 520 distinct elements, the length of the string arguments to a functions is limited to 5.The dataset contains around 100 thousand examples in total. A full description of the dataset can be found in Hupkes et al (2020).

A new English language dataset structured for task-oriented evaluation on unseen tasks. Source: Learning from Task Descriptions

Official dataset of Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP. See github.com/jsrozner/decrypt and https://doi.org/10.5061/dryad.n02v6wwzp

Suite of OpenAI Gym-compatible multi-agent reinforcement learning environment centered around meta-referential games to benchmark for behavioral traits pertaining to symbolic behaviours, as described in Santoro et al., 2021, "Symbolic Behaviours in Artificial Intelligence", with a primary focus on the following behavioural traits: * receptive, * constructive, * malleable, and * separable.