Offline RL

D4RL is a collection of environments for offline reinforcement learning. These environments include Maze2D, AntMaze, Adroit, Gym, Flow, FrankKitchen and CARLA. Source: https://sites.google.com/view/d4rl/home Image Source: https://sites.google.com/view/d4rl/home

An open database for sharing robotic experience, which provides an initial pool of 15 million video frames, from 7 different robot platforms, and study how it can be used to learn generalizable models for vision-based robotic manipulation. Source: RoboNet: Large-Scale Multi-Robot Learning

• NeoRL is a collection of environments and datasets for offline reinforcement learning with a special focus on real-world applications. The design follows real-world properties like the conservative of behavior policies, limited amounts of data, high-dimensional state and action spaces, and the highly stochastic nature of the environments. - The datasets include robotics, industrial control, finance trading and city management tasks with real-world properties, containing three-level sizes of dataset, three-level quality of data to mimic the dataset we will meet in offline RL scenarios. - Users can use the dataset to evaluate offline RL algorithms with near real-world application nature.

RL Unplugged is suite of benchmarks for offline reinforcement learning. The RL Unplugged is designed around the following considerations: to facilitate ease of use, the datasets are provided with a unified API which makes it easy for the practitioner to work with all data in the suite once a general pipeline has been established. This is a dataset accompanying the paper RL Unplugged: Benchmarks for Offline Reinforcement Learning. In this suite of benchmarks, the authors try to focus on the following problems: - High dimensional action spaces, for example the locomotion humanoid domains, there are 56 dimensional actions. - High dimensional observations. - Partial observability, observations have egocentric vision. - Difficulty of exploration, using states of the art algorithms and imitation to generate data for difficult environments. - Real world challenges. Source: DeepMind

RL Unplugged is suite of benchmarks for offline reinforcement learning. The RL Unplugged is designed around the following considerations: to facilitate ease of use, we provide the datasets with a unified API which makes it easy for the practitioner to work with all data in the suite once a general pipeline has been established. This is a dataset accompanying the paper RL Unplugged: Benchmarks for Offline Reinforcement Learning. In this suite of benchmarks, we try to focus on the following problems: High dimensional action spaces, for example the locomotion humanoid domains, we have 56 dimensional actions. High dimensional observations. Partial observability, observations have egocentric vision. Difficulty of exploration, using states of the art algorithms and imitation to generate data for difficult environments. Real world challenges.

This data contains about 2500 trajectories (with images and actions) of a Sawyer robot interacting with various objects. Examples from the dataset are shown in the adjacent video. We provide two versions of the VAL dataset - one with low-res images (1.4 GB) and one with high-res images (162 GB). The data quantity and format is the same between these two versions; the difference is only the image observation quality. The smaller dataset, with 48x48x3 images which can be used for eg. offline RL, is available for direct download: https://drive.google.com/file/d/1UuWANkVtWLg4egIK2LB_YCKuF87rMQ1H/view?usp=sharing The larger dataset, with 480x640x3 which might be preferred for eg. representation learning, is available at this Google drive folder: https://drive.google.com/drive/folders/1kD9kyP7-RlIrSnuN7rpEASAGWp5qnNov?usp=sharing To download the larger dataset, we suggest using https://rclone.org/ The data is sorted into several folders. There are a total of 300 files and 2500 trajectories. - fixed_drawer - Human-controlled demonstration data opening and closing drawers. (~10%) - fixed_pnp - Human-controlled demonstration data picking up objects. (~10%) - fixed_pot - Human-controlled demonstration data interacting with a pot and a lid. (~10%) - fixed_tray - Human-controlled demonstration data picking up objects and placing it in a tray. (~10%) - general - Further human-controlled demonstration data collected with the most diversity and variation. (~40%) - onpolicy_eval - Evaluation data collected by an RL policy. (~10%) - onpolicy_expl - Exploration data collected by an RL policy. (~10%)

NeoRL-2 includes new task scenarios that better reflect real-world task properties and includes traditional control methods as the data-collecting method. In summary, our contributions are as follows: 1. Tasks in NeoRL-2 cover a wider range of application domains, including robotics, aircraft, industrial pipelines, controllable nuclear fusion, healthcare, etc., encompassing key features such as delays, external factors, and safety constraints. 2. The data-collecting method in NeoRL-2 better aligns with real-world scenarios, employing deterministic methods for sampling. In some specific tasks, classical feedback controllers, such as Proportional-Integral-Derivative (PID) controller, are introduced. 3. We conducted experiments on these tasks using state-of-the-art (SOTA) offline RL algorithms and found that in most tasks, the trained policy of the current offline RL algorithms did not significantly outperform the behavior policy. By extending near real-world tasks in NeoRL-2, we hope that the development and implementation of RL in real-world scenarios can take into account these challenges and tackle more realistic domains.