General Reinforcement Learning

D4RL is a collection of environments for offline reinforcement learning. These environments include Maze2D, AntMaze, Adroit, Gym, Flow, FrankKitchen and CARLA. Source: https://sites.google.com/view/d4rl/home Image Source: https://sites.google.com/view/d4rl/home

Procgen Benchmark includes 16 simple-to-use procedurally-generated environments which provide a direct measure of how quickly a reinforcement learning agent learns generalizable skills.

An open database for sharing robotic experience, which provides an initial pool of 15 million video frames, from 7 different robot platforms, and study how it can be used to learn generalizable models for vision-based robotic manipulation. Source: RoboNet: Large-Scale Multi-Robot Learning

Obstacle Tower is a high fidelity, 3D, 3rd person, procedurally generated environment for reinforcement learning. An agent playing Obstacle Tower must learn to solve both low-level control and high-level planning problems in tandem while learning from pixels and a sparse reward signal. Unlike other benchmarks such as the Arcade Learning Environment, evaluation of agent performance in Obstacle Tower is based on an agent’s ability to perform well on unseen instances of the environment. Source: Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning

• NeoRL is a collection of environments and datasets for offline reinforcement learning with a special focus on real-world applications. The design follows real-world properties like the conservative of behavior policies, limited amounts of data, high-dimensional state and action spaces, and the highly stochastic nature of the environments. - The datasets include robotics, industrial control, finance trading and city management tasks with real-world properties, containing three-level sizes of dataset, three-level quality of data to mimic the dataset we will meet in offline RL scenarios. - Users can use the dataset to evaluate offline RL algorithms with near real-world application nature.

RL Unplugged is suite of benchmarks for offline reinforcement learning. The RL Unplugged is designed around the following considerations: to facilitate ease of use, the datasets are provided with a unified API which makes it easy for the practitioner to work with all data in the suite once a general pipeline has been established. This is a dataset accompanying the paper RL Unplugged: Benchmarks for Offline Reinforcement Learning. In this suite of benchmarks, the authors try to focus on the following problems: - High dimensional action spaces, for example the locomotion humanoid domains, there are 56 dimensional actions. - High dimensional observations. - Partial observability, observations have egocentric vision. - Difficulty of exploration, using states of the art algorithms and imitation to generate data for difficult environments. - Real world challenges. Source: DeepMind

Avalon is a benchmark for generalization in Reinforcement Learning (RL). The benchmark consists of a set of tasks in which embodied agents in highly diverse procedural 3D worlds must survive by navigating terrain, hunting or gathering food, and avoiding hazards. Avalon is unique among existing RL benchmarks in that the reward function, world dynamics, and action space are the same for every task, with tasks differentiated solely by altering the environment; its 20 tasks, ranging in complexity from eat and throw to hunt and navigate, each create worlds in which the agent must perform specific skills in order to survive. This benchmark setup enables investigations of generalization within tasks, between tasks, and to compositional tasks that require combining skills learned from previous tasks. Source: Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds Image Source: https://arxiv.org/pdf/2210.13417v1.pdf

The ability to jointly understand the geometry of objects and plan actions for manipulating them is crucial for intelligent agents. This ability is referred to as geometric planning. Recently, many interactive environments have been proposed to evaluate intelligent agents on various skills, however, none of them cater to the needs of geometric planning. PackIt is a virtual environment to evaluate and potentially learn the ability to do geometric planning, where an agent needs to take a sequence of actions to pack a set of objects into a box with limited space.

POPGym is designed to benchmark memory in deep reinforcement learning. It contains a set of environments and a collection of memory model baselines. The environments are all Partially Observable Markov Decision Process (POMDP) environments following the Openai Gym interface. Our environments follow a few basic tenets: 1. Painless Setup - popgym environments require only gym, numpy, and mazelib as dependencies 2. Laptop-Sized Tasks - Most tasks can be solved in less than a day on the CPU 3. True Generalization - All environments are heavily randomized. The paper uses 15M environment steps for each trial.

WildLife Documentary is an animal object detection dataset. It contains 15 documentary films that are downloaded from YouTube. The videos vary between 9 minutes to as long as 50 minutes, with resolution ranging from 360p to 1080p. A unique property of this dataset is that all videos are accompanied with subtitles that are automatically generated from speech by YouTube. The subtitles are revised manually to correct obvious spelling mistakes. All the animals in the videos are annotated, resulting in more than 4098 object tracklets of 60 different visual concepts, e.g., ‘tiger’, ‘koala’, ‘langur’, and ‘ostrich’.

The bipedal skills benchmark is a suite of reinforcement learning environments implemented for the MuJoCo physics simulator. It aims to provide a set of tasks that demand a variety of motor skills beyond locomotion, and is intended for evaluating skill discovery and hierarchical learning methods. The majority of tasks exhibit a sparse reward structure.

RL Unplugged is suite of benchmarks for offline reinforcement learning. The RL Unplugged is designed around the following considerations: to facilitate ease of use, we provide the datasets with a unified API which makes it easy for the practitioner to work with all data in the suite once a general pipeline has been established. This is a dataset accompanying the paper RL Unplugged: Benchmarks for Offline Reinforcement Learning. In this suite of benchmarks, we try to focus on the following problems: High dimensional action spaces, for example the locomotion humanoid domains, we have 56 dimensional actions. High dimensional observations. Partial observability, observations have egocentric vision. Difficulty of exploration, using states of the art algorithms and imitation to generate data for difficult environments. Real world challenges.

This data contains about 2500 trajectories (with images and actions) of a Sawyer robot interacting with various objects. Examples from the dataset are shown in the adjacent video. We provide two versions of the VAL dataset - one with low-res images (1.4 GB) and one with high-res images (162 GB). The data quantity and format is the same between these two versions; the difference is only the image observation quality. The smaller dataset, with 48x48x3 images which can be used for eg. offline RL, is available for direct download: https://drive.google.com/file/d/1UuWANkVtWLg4egIK2LB_YCKuF87rMQ1H/view?usp=sharing The larger dataset, with 480x640x3 which might be preferred for eg. representation learning, is available at this Google drive folder: https://drive.google.com/drive/folders/1kD9kyP7-RlIrSnuN7rpEASAGWp5qnNov?usp=sharing To download the larger dataset, we suggest using https://rclone.org/ The data is sorted into several folders. There are a total of 300 files and 2500 trajectories. - fixed_drawer - Human-controlled demonstration data opening and closing drawers. (~10%) - fixed_pnp - Human-controlled demonstration data picking up objects. (~10%) - fixed_pot - Human-controlled demonstration data interacting with a pot and a lid. (~10%) - fixed_tray - Human-controlled demonstration data picking up objects and placing it in a tray. (~10%) - general - Further human-controlled demonstration data collected with the most diversity and variation. (~40%) - onpolicy_eval - Evaluation data collected by an RL policy. (~10%) - onpolicy_expl - Exploration data collected by an RL policy. (~10%)

NeoRL-2 includes new task scenarios that better reflect real-world task properties and includes traditional control methods as the data-collecting method. In summary, our contributions are as follows: 1. Tasks in NeoRL-2 cover a wider range of application domains, including robotics, aircraft, industrial pipelines, controllable nuclear fusion, healthcare, etc., encompassing key features such as delays, external factors, and safety constraints. 2. The data-collecting method in NeoRL-2 better aligns with real-world scenarios, employing deterministic methods for sampling. In some specific tasks, classical feedback controllers, such as Proportional-Integral-Derivative (PID) controller, are introduced. 3. We conducted experiments on these tasks using state-of-the-art (SOTA) offline RL algorithms and found that in most tasks, the trained policy of the current offline RL algorithms did not significantly outperform the behavior policy. By extending near real-world tasks in NeoRL-2, we hope that the development and implementation of RL in real-world scenarios can take into account these challenges and tackle more realistic domains.