Robot Manipulation

RLBench is an ambitious large-scale benchmark and learning environment designed to facilitate research in a number of vision-guided manipulation research areas, including: reinforcement learning, imitation learning, multi-task learning, geometric computer vision, and in particular, few-shot learning.

CALVIN (Composing Actions from Language and Vision), is an open-source simulated benchmark to learn long-horizon language-conditioned robot manipulation tasks.

ManiSkill2 is the next generation of the SAPIEN ManiSkill benchmark, to address critical pain points often encountered by researchers when using benchmarks for generalizable manipulation skills. It includes 20 manipulation task families with 2000+ object models and 4M+ demonstration frames, which cover stationary/mobile-base, single/dual-arm, and rigid/soft-body manipulation tasks with 2D/3D input data simulated by fully dynamic engines. Source: ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills Image Source: https://arxiv.org/pdf/2302.04659v1.pdf

To realize effective large-scale, real-world robotic applications, we must evaluate how well our robot policies adapt to changes in environmental conditions. Unfortunately, a majority of studies evaluate robot performance in environments closely resembling or even identical to the training setup. We present Colosseum, a novel simulation benchmark,with 20 diverse manipulation tasks, that enables systematical evaluation of models across 12 axes of environmental perturbations. These perturbations include changes in color, texture, and size of objects, table-tops, and backgrounds; we also vary lighting, distractors, and camera pose. Using Colosseum, we compare 4 state-of-the-art manipulation models to reveal that their success rate degrades between 30-50% across these perturbation factors. When multiple perturbations are applied in unison, the success rate degrades > 75%. We identify that changing the number of distractor objects, target object color, or lighting conditions are the perturbations that reduce model performance the most. To verify the ecological validity of our results, we show that our results in simulation are correlated (R2 = 0.614) to similar perturbations in real-world experiments. We open source code for others to use Colosseum, and also release code to 3D print the objects used to replicate the real-world perturbations. Ultimately, we hope that Colosseum will serve as a benchmark to identify modeling decisions that systematically improve generalization for manipulation.

Significant progress has been made in building generalist robot manipulation policies, yet their scalable and reproducible evaluation remains challenging, as real-world evaluation is operationally expensive and inefficient. We propose employing physical simulators as efficient, scalable, and informative complements to real-world evaluations. These simulation evaluations offer valuable quantitative metrics for checkpoint selection, insights into potential real-world policy behaviors or failure modes, and standardized setups to enhance reproducibility. This repository's code is based in the SAPIEN simulator and the CPU based ManiSkill2 benchmark. We have also integrated the Bridge dataset environments into ManiSkill3, which offers GPU parallelization and can run 10-15x faster than the ManiSkill2 version. For instructions on how to use the GPU parallelized environments and evaluate policies on them, see: https://github.com/simpler-env/SimplerEnv/tree/maniskill3 This repository encompasses 2 real-to-sim evaluation setups: Visual Matching evaluation: Matching real & sim visual appearances for policy evaluation by overlaying real-world images onto simulation backgrounds and adjusting foreground object and robot textures in simulation. Variant Aggregation evaluation: creating different sim environment variants (e.g., different backgrounds, lightings, distractors, table textures, etc) and averaging their results.

Significant progress has been made in building generalist robot manipulation policies, yet their scalable and reproducible evaluation remains challenging, as real-world evaluation is operationally expensive and inefficient. We propose employing physical simulators as efficient, scalable, and informative complements to real-world evaluations. These simulation evaluations offer valuable quantitative metrics for checkpoint selection, insights into potential real-world policy behaviors or failure modes, and standardized setups to enhance reproducibility. This repository's code is based in the SAPIEN simulator and the CPU based ManiSkill2 benchmark. We have also integrated the Bridge dataset environments into ManiSkill3, which offers GPU parallelization and can run 10-15x faster than the ManiSkill2 version. For instructions on how to use the GPU parallelized environments and evaluate policies on them, see: https://github.com/simpler-env/SimplerEnv/tree/maniskill3 This repository encompasses 2 real-to-sim evaluation setups: Visual Matching evaluation: Matching real & sim visual appearances for policy evaluation by overlaying real-world images onto simulation backgrounds and adjusting foreground object and robot textures in simulation. Variant Aggregation evaluation: creating different sim environment variants (e.g., different backgrounds, lightings, distractors, table textures, etc) and averaging their results. We hope that our work guides and inspires future real-to-sim evaluation efforts.

A Modular Simulation Framework and Benchmark for Robot Learning.

The YCB-Slide dataset comprises of DIGIT sliding interactions on YCB objects. We envision this can contribute towards efforts in tactile localization, mapping, object understanding, and learning dynamics models. We provide access to DIGIT images, sensor poses, RGB video feed, ground-truth mesh models, and ground-truth heightmaps + contact masks (simulation only). This dataset is supplementary to the MidasTouch paper, a CoRL 2022 submission.

• For each BDLO, dynamic trajectory data is captured in real-world settings using a motion capture system operating at 100 Hz when robots grasp the BDLO’s ends. For details on dataset usage, please refer to DEFT_train.py. - For BDLO 1 and BDLO 3, we record dynamic trajectory data when one robot grasps the middle of the BDLO while the other robot grasps one of its ends.

For each DLO, we collect 350 seconds of dynamic trajectory data in the real-world using the motion capture system at a frequency of 100 Hz.

Two versions of the dataset are offered: one is the full dataset used to train the models in DeformPAM, and the other is a mini dataset for easier examination. Both datasets include data for the supervised and finetuning stages of granular pile shaping, rope shaping, and T-shirt unfolding.

Demonstration data for 4 FurnitureBench tasks collected with a SpaceMouse using a DiffIK Controller. Tasks: one_leg, round_table, lamp, and square_table. Can be used to train policies from human data in simulation, as opposed to the scripted data included with the benchmark.

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

In this dataset we teleoperated UR5 arm to collect manipulation data for picking up a screwdriver in a cluttered tabletop environment.

Two versions of the dataset are offered: one is the full dataset used to train the models in our paper, and the other is a mini dataset for easier examination. Both versions include raw and postprocessed subsets of peeling, wiping and lifting. The raw videos of the tactile dataset used for generate the PCA embedding are also provided.

From https://github.com/MMintLab/VIRDO/blob/master/data/dataset_readme.txt, 1. DESCRIPTION: This dataset is written in 'dtype=torch.float64'. This dataset consists of total 144 deformation scenes from 6 different objects generated through MATLAB. It is divided into 'train' and 'test' dataset, where data['train'][OBJECT IDX = i][DEFORM IDX = j ] and data['test'][OBJECT IDX = i][DEFORM IDX = j ] indicates the same scene, but they are two different subsets of query points. 2. STRUCTURE: The dataset structure is as follows: VIRDO_simul_dataset = { 'train':{ <OBJECT IDX>: { 'nominal': { 'coords': tensor([1, M, 3]), 'normals': tensor([1, M, 3]), 'gt': tensor([1, M, 3]), 'scale': float }, <DEFORM IDX>: { 'coords': tensor([1, M, 3]), 'contact': tensor([1, M_c, 3]), 'normals': tensor([1, M, 3]), 'gt': tensor([1, M, 3]), 'scale': float, 'reaction': tensor([1,3] }, }, }, 'test':{ <OBJECT IDX>: { 'nominal': { 'coords': tensor([1, M, 3]), 'normals': tensor([1, M, 3]), 'gt': tensor([1, M, 3]), 'scale': float }, <DEFORM IDX>: { 'coords': tensor([1, M, 3]), 'contact': tensor([1, M_c, 3]), 'normals': tensor([1, M, 3]), 'gt': tensor([1, M, 3]), 'scale': float, 'reaction': tensor([1,3] }, }, }, } * <OBJECT IDX> = Interger from 0 ~ 5. Each number indicates different object. * <DEFORM IDX> = Unique integer for each deformation. * M = total points (on-surface + off-surface) * M_c = a subset of on-surface points that are in contact * [:,i,:] elements of 'coords', 'normals', and 'gt' refers ith query point of a scene. To get on-surface points of data_def = data['train'][<OBJECT IDX>][<DEFORM IDX>], you should do data_def['coords'][:,torch.where(data_def['gt'] == 0)[1],:].