Procgen Benchmark includes 16 simple-to-use procedurally-generated environments which provide a direct measure of how quickly a reinforcement learning agent learns generalizable skills.

PRM800K is a process supervision dataset containing 800,000 step-level correctness labels for model-generated solutions to problems from the MATH dataset. Source: PRM800K: A Process Supervision Dataset | GitHub Repo Image Source: PRM800K: A Process Supervision Dataset | GitHub Repo

ManiSkill2 is the next generation of the SAPIEN ManiSkill benchmark, to address critical pain points often encountered by researchers when using benchmarks for generalizable manipulation skills. It includes 20 manipulation task families with 2000+ object models and 4M+ demonstration frames, which cover stationary/mobile-base, single/dual-arm, and rigid/soft-body manipulation tasks with 2D/3D input data simulated by fully dynamic engines. Source: ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills Image Source: https://arxiv.org/pdf/2302.04659v1.pdf

SMACv2 (StarCraft Multi-Agent Challenge v2) is a new version of the benchmark where scenarios are procedurally generated and require agents to generalise to previously unseen settings (from the same distribution) during evaluation. Source: SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning Image Source: https://github.com/oxwhirl/smacv2

V-D4RL provides pixel-based analogues of the popular D4RL benchmarking tasks, derived from the dm_control suite, along with natural extensions of two state-of-the-art online pixel-based continuous control algorithms, DrQ-v2 and DreamerV2, to the offline setting.

This is a gun detection dataset with 51K annotated gun images for gun detection and other 51K cropped gun chip images for gun classification collected from a few different sources.

QDax is a benchmark suite designed for for Deep Neuroevolution in Reinforcement Learning domains for robot control. The suite includes the definition of tasks, environments, behavioral descriptors, and fitness. It specify different benchmarks based on the complexity of both the task and the agent controlled by a deep neural network. The benchmark uses standard Quality-Diversity metrics, including coverage, QD-score, maximum fitness, and an archive profile metric to quantify the relation between coverage and fitness. Source: Benchmarking Quality-Diversity Algorithms on Neuroevolution for Reinforcement Learning Image Source: https://arxiv.org/pdf/2211.02193v1.pdf

Avalon is a benchmark for generalization in Reinforcement Learning (RL). The benchmark consists of a set of tasks in which embodied agents in highly diverse procedural 3D worlds must survive by navigating terrain, hunting or gathering food, and avoiding hazards. Avalon is unique among existing RL benchmarks in that the reward function, world dynamics, and action space are the same for every task, with tasks differentiated solely by altering the environment; its 20 tasks, ranging in complexity from eat and throw to hunt and navigate, each create worlds in which the agent must perform specific skills in order to survive. This benchmark setup enables investigations of generalization within tasks, between tasks, and to compositional tasks that require combining skills learned from previous tasks. Source: Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds Image Source: https://arxiv.org/pdf/2210.13417v1.pdf

FinRL-Meta is universe of market environments for data-driven financial reinforcement learning. It follows the de facto standard of OpenAI Gym and the lean principle of software development. It has the following unique features of layered structure and extensibility, training-testing-trading pipeline and plug-and-play mode. Source: FinRL-Meta: Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning Image Source: https://arxiv.org/pdf/2211.03107v1.pdf

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

POPGym is designed to benchmark memory in deep reinforcement learning. It contains a set of environments and a collection of memory model baselines. The environments are all Partially Observable Markov Decision Process (POMDP) environments following the Openai Gym interface. Our environments follow a few basic tenets: 1. Painless Setup - popgym environments require only gym, numpy, and mazelib as dependencies 2. Laptop-Sized Tasks - Most tasks can be solved in less than a day on the CPU 3. True Generalization - All environments are heavily randomized. The paper uses 15M environment steps for each trial.

MIDGARD is an open-source simulator for autonomous robot navigation in outdoor unstructured environments. It is designed to enable the training of autonomous agents (e.g., unmanned ground vehicles) in photorealistic 3D environments, and support the generalization skills of learning-based agents thanks to the variability in training scenarios.

Synthetic OD data to mimic data showed in the application of the paper.

first everyday task dataset featuring COT outputs, diverse task designs, detailed re-plan processes, along with SFT and DPO sub-datasets.

lilGym is a benchmark for language-conditioned reinforcement learning in visual environment based on 2,661 highly-compositional human-written natural language statements grounded in an interactive visual environment. Each statement is paired with multiple start states and reward functions to form thousands of distinct Markov Decision Processes of varying difficulty. Source: lilGym: Natural Language Visual Reasoning with Reinforcement Learning Image Source: https://github.com/lil-lab/lilgym

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

The dataset comprises 1641 questions and answers generated as three separate parts. The first part of the dataset contains questions and answers that test the model’s ability to understand the current status of the environment and the provided rules. The second part of the dataset contains questions that test the model’s ability to generate valid actions, both in terms of syntax (JSON format) and semantics (validity in the specific state). The 3rd part of the dataset part aims to teach the fine-tuned models how to make correct decisions given a specific environment state. The dataset was used to fine-tune LLMs to perform tasks in the NetSecGame environment.

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

PushWorld is an environment with simplistic physics that requires manipulation planning with both movable obstacles and tools. It contains more than 200 PushWorld puzzles in PDDL and in an OpenAI Gym environment. Source: PushWorld: A benchmark for manipulation planning with tools and movable obstacles

The Room environment - v0 DOI We have released a challenging Gymnasium compatible environment. The best strategy for this environment is to have both episodic and semantic memory systems. See the paper for more information. ## Prerequisites 1. A unix or unix-like x86 machine 1. python 3.10 or higher. 1. Running in a virtual environment (e.g., conda, virtualenv, etc.) is highly recommended so that you don't mess up with the system python. 1. This env is added to the PyPI server. Just run: pip install room-env ## Data collection Data is collected from querying ConceptNet APIs. For simplicity, we only collect triples whose format is (head, atlocation, tail). Here head is one of the 80 MS COCO dataset categories. This was kept in mind so that later on we can use images as well. If you want to collect the data manually, then run below: python collect_data.py ## How does this environment work? The Gymnasium-compatible Room environment is one big room with N_people number of people who can freely move around. Each of them selects one object, among N_objects, and places it in one of the N_locations locations. N_agents number of agent(s) are also in this room. They can only observe one human placing an object, one at a time; x^(t). At the same time, they are given one question about the location of an object; q^(t). x^(t) is given as a quadruple, (h^(t),r^(t),t^(t),t), For example, <James’s laptop, atlocation, James’s desk, 42> accounts for an observation where an agent sees James placing his laptop on his desk at t = 42. q^(t) is given as a double, (h,r). For example, <Karen’s cat, atlocation> is asking where Karen’s cat is located. If the agent answers the question correctly, it gets a reward of + 1, and if not, it gets 0. The reason why the observations and questions are given as RDF-triple-like format is two folds. One is that this structured format is easily readable / writable by both humans and machines. Second is that we can use existing knowledge graphs, such as ConceptNet . To simplify the environment, the agents themselves are not actually moving, but the room is continuously changing. There are several random factors in this environment to be considered: 1. With the chance of p_commonsense, a human places an object in a commonsense location (e.g., a laptop on a desk). The commonsense knowledge we use is from ConceptNet. With the chance of 1 − p_commonsense, an object is placed at a non-commonsense random location (e.g., a laptop on the tree). 1. With the chance of p_{new_location}, a human changes object location. 1. With the chance of p_{new_object}, a human changes his/her object to another one. 1. With the chance of p_{switch_person}, two people switch their locations. This is done to mimic an agent moving around the room. All of the four probabilities account for the Bernoulli distributions. Consider there is only one agent. Then this is a POMDP, where S_t = (x^(t), q^(t)), A_t = (do something with x^(t), answer q^(t)), and R_t ∈ {0, 1}. Currently there is no RL trained for this. We only have some heuristics. Take a look at the paper for more details. ## RoomEnv-v0 python import gymnasium as gym env = gym.make("room_env:RoomEnv-v0") (observation, question), info = env.reset() rewards = 0 while True: (observation, question), reward, done, truncated, info = env.step("This is my answer!") rewards += reward if done: break print(rewards) Every time when an agent takes an action, the environment will give you an observation and a question to answer. You can try directly answering the question, such as env.step("This is my answer!"), but a better strategy is to keep the observations in memory systems and take advantage of the current observation and the history of them in the memory systems. Take a look at this repo for an actual interaction with this environment to learn a policy. ## Contributing Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated. 1. Fork the Project 1. Create your Feature Branch (git checkout -b feature/AmazingFeature) 1. Run make test && make style && make quality in the root repo directory, to ensure code quality. 1. Commit your Changes (git commit -m 'Add some AmazingFeature') 1. Push to the Branch (git push origin feature/AmazingFeature) 1. Open a Pull Request ## Cite our paper bibtex @misc{https://doi.org/10.48550/arxiv.2204.01611, doi = {10.48550/ARXIV.2204.01611}, url = {https://arxiv.org/abs/2204.01611}, author = {Kim, Taewoon and Cochez, Michael and Francois-Lavet, Vincent and Neerincx, Mark and Vossen, Piek}, keywords = {Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences}, title = {A Machine With Human-Like Memory Systems}, publisher = {arXiv}, year = {2022}, copyright = {Creative Commons Attribution 4.0 International} } ## Authors - Taewoon Kim - Michael Cochez - Vincent Francois-Lavet - Mark Neerincx - Piek Vossen ## License MIT

The Room environment - v1 DOI We have released a challenging Gymnasium compatible environment. The best strategy for this environment is to have both episodic and semantic memory systems. See the paper for more information. ## Prerequisites 1. A unix or unix-like x86 machine 1. python 3.10 or higher. 1. Running in a virtual environment (e.g., conda, virtualenv, etc.) is highly recommended so that you don't mess up with the system python. 1. This env is added to the PyPI server. Just run: pip install room-env ## RoomEnv-v1 python import gymnasium as gym env = gym.make("room_env:RoomEnv-v1") (observation, question), info = env.reset() rewards = 0 while True: # There is one different thing in the RoomEnv from the original AAAI-2023 paper: # The reward is either +1 or -1, instead of +1 or 0. (observation, question), reward, done, truncated, info = env.step("This is my answer!") rewards += reward if done: break json { "des_size": "l", "seed": 42, "question_prob": 1.0, "allow_random_human": False, "allow_random_question": False, "total_maximum_episode_rewards": 128, "check_resources": True, } If you want to create an env with a different set of parameters, you can do so. For example: python env_params = {"seed": 0, "allow_random_human": True, "pretrain_semantic": True} env = gym.make("room_env:RoomEnv-v1", **env_params) Take a look at this repo for an actual interaction with this environment to learn a policy. ## Data collection Data is collected from querying ConceptNet APIs. For simplicity, we only collect triples whose format is (head, atlocation, tail). Here head is one of the 80 MS COCO dataset categories. This was kept in mind so that later on we can use images as well. If you want to collect the data manually, then run below: python collect_data.py ## The RoomDes The DES is part of RoomEnv. You don't have to care about how it works. If you are still curious, you can read below. You can run the RoomDes by python from room_env.des import RoomDes des = RoomDes() des.run(debug=True) with debug=True it'll print events (i.e., state changes) to the console. console {'resource_changes': {'desk': -1, 'lap': 1}, 'state_changes': {'Vincent': {'current_time': 1, 'object_location': {'current': 'desk', 'previous': 'lap'}}}} {'resource_changes': {}, 'state_changes': {}} {'resource_changes': {}, 'state_changes': {}} {'resource_changes': {}, 'state_changes': {'Michael': {'current_time': 4, 'object_location': {'current': 'lap', 'previous': 'desk'}}, 'Tae': {'current_time': 4, 'object_location': {'current': 'desk', 'previous': 'lap'}}}} ## Contributing Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated. 1. Fork the Project 1. Create your Feature Branch (git checkout -b feature/AmazingFeature) 1. Run make test && make style && make quality in the root repo directory, to ensure code quality. 1. Commit your Changes (git commit -m 'Add some AmazingFeature') 1. Push to the Branch (git push origin feature/AmazingFeature) 1. Open a Pull Request ## Cite our paper bibtex @article{Kim_Cochez_Francois-Lavet_Neerincx_Vossen_2023, title={A Machine with Short-Term, Episodic, and Semantic Memory Systems}, volume={37}, url={https://ojs.aaai.org/index.php/AAAI/article/view/25075}, DOI={10.1609/aaai.v37i1.25075}, abstractNote={Inspired by the cognitive science theory of the explicit human memory systems, we have modeled an agent with short-term, episodic, and semantic memory systems, each of which is modeled with a knowledge graph. To evaluate this system and analyze the behavior of this agent, we designed and released our own reinforcement learning agent environment, “the Room”, where an agent has to learn how to encode, store, and retrieve memories to maximize its return by answering questions. We show that our deep Q-learning based agent successfully learns whether a short-term memory should be forgotten, or rather be stored in the episodic or semantic memory systems. Our experiments indicate that an agent with human-like memory systems can outperform an agent without this memory structure in the environment.}, number={1}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Kim, Taewoon and Cochez, Michael and Francois-Lavet, Vincent and Neerincx, Mark and Vossen, Piek}, year={2023}, month={Jun.}, pages={48-56} } ## Authors - Taewoon Kim - Michael Cochez - Vincent Francois-Lavet - Mark Neerincx - Piek Vossen ## License MIT

The Room environment - v2 DOI We have released a challenging Gymnasium compatible environment. See the paper for more information. <img src="figures/room-layout-xl.png" alt="" style="width: 50%;"/><img src="figures/room-layout-kg-xl.png" alt="" style="width: 50%;"/> ## Prerequisites 1. A unix or unix-like x86 machine 1. python 3.10 or higher. 1. Running in a virtual environment (e.g., conda, virtualenv, etc.) is highly recommended so that you don't mess up with the system python. 1. This env is added to the PyPI server. Just run: pip install room-env ## Creating a RoomEnv-v2 python import random from room_env.create_room_v2 import RoomCreator room_creator = RoomCreator( filename="dev", grid_length=7, num_rooms=32, num_static_objects=8, num_independent_objects=8, num_dependent_objects=8, room_prob=0.5, minimum_transition_stay_prob=0.6, static_object_in_every_room=False, give_fake_names=False, ) room_creator.run() ./room-env-v2.ipynb has some good examples. ## Running a RoomEnv-v2 python import gymnasium as gym import random env = gym.make("room_env:RoomEnv-v2", room_size="l") observations, info = env.reset() rewards = 0 while True: observations, reward, done, truncated, info = env.step( ( ["random answer"] * len(observations["questions"]), random.choice(["north", "east", "south", "west", "stay"]), ) ) rewards += reward if done or truncated: break # You can also get the map of the rooms room_layout = env.unwrapped.return_room_layout(exclude_walls=True) Take a look at this repo for an actual interaction with this environment to learn a policy. ## Contributing Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated. 1. Fork the Project 1. Create your Feature Branch (git checkout -b feature/AmazingFeature) 1. Run make test && make style && make quality in the root repo directory, to ensure code quality. 1. Commit your Changes (git commit -m 'Add some AmazingFeature') 1. Push to the Branch (git push origin feature/AmazingFeature) 1. Open a Pull Request ## Cite our paper bibtex @misc{kim2024leveragingknowledgegraphbasedhumanlike, title={Leveraging Knowledge Graph-Based Human-Like Memory Systems to Solve Partially Observable Markov Decision Processes}, author={Taewoon Kim and Vincent François-Lavet and Michael Cochez}, year={2024}, eprint={2408.05861}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2408.05861}, } ## Authors - Taewoon Kim - Michael Cochez - Vincent Francois-Lavet ## License MIT