Robot Navigation

TAO is a federated dataset for Tracking Any Object, containing 2,907 high resolution videos, captured in diverse environments, which are half a minute long on average. A bottom-up approach was used for discovering a large vocabulary of 833 categories, an order of magnitude more than prior tracking benchmarks. The dataset was annotated by labelling tracks for objects that move at any point in the video, and giving names to them post factum. Source: TAO: A Large-Scale Benchmark for Tracking Any Object

Gibson is an opensource perceptual and physics simulator to explore active and real-world perception. The Gibson Environment is used for Real-World Perception Learning.

A platform for research in embodied artificial intelligence (AI). Source: Habitat: A Platform for Embodied AI Research

QDax is a benchmark suite designed for for Deep Neuroevolution in Reinforcement Learning domains for robot control. The suite includes the definition of tasks, environments, behavioral descriptors, and fitness. It specify different benchmarks based on the complexity of both the task and the agent controlled by a deep neural network. The benchmark uses standard Quality-Diversity metrics, including coverage, QD-score, maximum fitness, and an archive profile metric to quantify the relation between coverage and fitness. Source: Benchmarking Quality-Diversity Algorithms on Neuroevolution for Reinforcement Learning Image Source: https://arxiv.org/pdf/2211.02193v1.pdf

A dataset with high resolution (4K) images and manually-annotated dense labels every 50 frames. Source: Semantics through Time: Semi-supervised Segmentation of Aerial Videos with Iterative Label Propagation

BASEPROD provides comprehensive rover sensor data collected over a 1.7 km traverse, accompanied by high-resolution 2D and 3D drone maps of the terrain. The dataset also includes laser-induced breakdown spectroscopy (LIBS) measurements from key sampling sites along the rover's path, as well as weather station data to contextualize environmental conditions. The rover is a half-scale model of the ExoMars rover, featuring a sophisticated six-wheel triple-bogie suspension system. Each wheel assembly includes three actuators— one for wheel deployment (see e.g., wheel-walking), one for steering, and one for driving — and is equipped with a force-torque sensor. The sensor suite also comprises an RGB-D camera, a thermal camera, and a stereo camera mounted on a mast with pan-tilt capability. Additionally, the dataset contains IMU, GNSS, and gyroscope measurements, facilitating comprehensive analysis. Potential Applications: Research in navigation, localization, mapping, odometry, and terrain classification or segmentation tasks. It provides a rich resource for developing and testing algorithms in planetary exploration conditions.

A benchmark for detecting fallen people lying on the floor. It consists of 6982 images, with a total of 5023 falls and 2275 non falls corresponding to people in conventional situations (standing up, sitting, lying on the sofa or bed, walking, etc). Almost all the images have been captured in indoor environments with very different situations: variation of poses and sizes, occlusions, lighting changes, etc. Source: FPDS

MIDGARD is an open-source simulator for autonomous robot navigation in outdoor unstructured environments. It is designed to enable the training of autonomous agents (e.g., unmanned ground vehicles) in photorealistic 3D environments, and support the generalization skills of learning-based agents thanks to the variability in training scenarios.

Sparrow-V0: A Reinforcement Learning Friendly Simulator for Mobile Robot Features: Vectorizable (Enable fast data collection; Single environment is also supported) Domain Randomization (control interval, control delay, maximum velocity, inertia, friction, the magnitude of sensor noise and maps can be randomized while training) Lightweight (Consume only 150~200 mb RAM or GPU memories per environment) Standard Gym API with both Pytorch/Numpy data flow GPU/CPU are both acceptable (If you use Pytorch to build your RL model, you can run your RL model and Sparrow both on GPU. Then you don't need to transfer the transitions from CPU to GPU anymore.) Easy to use (30kb pure Python files. Just import, never worry about installation) Ubuntu/Windows are both supported Accept image as map (Customize your own environments easily and rapidly) Detailed comments on source code.

HabiCrowd, a new dataset and benchmark for crowd-aware visual navigation that surpasses other benchmarks in terms of human diversity and computational utilization. HabiCrowd can be utilized to study crowd-aware visual navigation tasks. A notable feature of HabiCrowd is that our crowd-aware settings is 3D, which is scarcely studied by previous works.

ISOD contains 2,000 manually labelled RGB-D images from 20 diverse sites, each featuring over 30 types of small objects randomly placed amidst the items already present in the scenes. These objects, typically ≤3cm in height, include LEGO blocks, rags, slippers, gloves, shoes, cables, crayons, chalk, glasses, smartphones (and their cases), fake banana peels, fake pet waste, and piles of toilet paper, among others. These items were chosen because they either threaten the safe operation of indoor mobile robots or create messes if run over. In addition to RGB images, ISOD also includes corresponding depth images and IMU readings. A reference image of each floor type was also recorded using a smartphone. This dataset was used as a real-world validation dataset in the original work to explore the performance of the model beyond synthetic data, specifically focusing on the potential application of real-time robot navigation.

MlGesture is a dataset for hand gesture recognition tasks, recorded in a car with 5 different sensor types at two different viewpoints. The dataset contains over 1300 hand gesture videos from 24 participants and features 9 different hand gesture symbols. One sensor cluster with five different cameras is mounted in front of the driver in the center of the dashboard. A second sensor cluster is mounted on the ceiling looking straight down. Source: MLGESTURE DATASET

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

In the last years, the research interest in visual navigation towards objects in indoor environments has grown significantly. This growth can be attributed to the recent availability of large navigation datasets in photo-realistic simulated environments, like Gibson and Matterport3D. However, the navigation tasks supported by these datasets are often restricted to the objects present in the environment at acquisition time. Also, they fail to account for the realistic scenario in which the target object is a user-specific instance that can be easily confused with similar objects and may be found in multiple locations within the environment. To address these limitations, we propose a new task denominated Personalized Instance-based Navigation (PIN), in which an embodied agent is tasked with locating and reaching a specific personal object by distinguishing it among multiple instances of the same category. The task is accompanied by PInNED, a dedicated new dataset composed of photo-realistic scenes augmented with additional 3D objects. In each episode, the target object is presented to the agent using two modalities: a set of visual reference images on a neutral background and manually annotated textual descriptions. Through comprehensive evaluations and analyses, we showcase the challenges of the PIN task as well as the performance and shortcomings of currently available methods designed for object-driven navigation, considering modular and end-to-end agents.

SEmantic Salient Instance Video (SESIV) dataset is obtained by augmenting the DAVIS-2017 benchmark dataset by assigning semantic ground-truth for salient instance labels. The SESIV dataset consists of 84 high-quality video sequences with pixel-wisely per-frame ground-truth labels. Source: Semantic Instance Meets Salient Object: Study on Video Semantic Salient Instance Segmentation

A dataset collected in a set of experiments that involves human participants and a robot. Source: SFU-Store-Nav: A Multimodal Dataset for Indoor Human Navigation

The scene derives from photo-realistic HM3D datasets. Our dataset offers a wide variety of environments especially for Social Navigation tasks, with carefully calibrated human density, incorporating realistic human motions and natural movement patterns. These features ensure balanced interaction dynamics across diverse scenes, facilitating the development of more effective social navigation algorithms.

The scene derives from photo-realistic MP3D datasets. Our dataset offers a wide variety of environments especially for Social Navigation tasks, with carefully calibrated human density, incorporating realistic human motions and natural movement patterns. These features ensure balanced interaction dynamics across diverse scenes, facilitating the development of more effective social navigation algorithms.

This dataset involves a 2D or 3D agent moving from a start to goal pose while interacting with nearby objects. These objects can influence position of the agent via attraction or repulsion forces as well as influence orientation via attraction to object's orientation. This dataset can be used to pre-train general policy behavior, which can be later fine-tuned quickly for a person's specific preferences. Example use-cases include: - self-driving cars maintaining distance from other cars - robot pick-and-place tasks with intermediate subtasks (ie: scanning factory items before dropping them off) Overall, pre-training initial policy behavior to be fine-tuned later is a powerful paradigm and is arguably essential for robots to handle changing environments and user preferences. This is compared to the paradigm of training on massive amounts of data and remaining fixed at test time, hoping that generalization alone will help the agent handle new scenarios.

Introduction This dataset was gathered during the Vid2RealHRI study of humans’ perception of robots' intelligence in the context of an incidental Human-Robot encounter. The dataset contains participants' questionnaire responses to four video study conditions, namely Baseline, Verbal, Body language, and Body language + Verbal. The videos depict a scenario where a pedestrian incidentally encounters a quadruped robot trying to enter a building. The robot uses verbal commands or body language to try to ask for help from the pedestrian in different study conditions. The differences in the conditions were manipulated using the robot’s verbal and expressive movement functionalities. ## Dataset Purpose The dataset includes the responses of human subjects about the robots' social intelligence used to validate the hypothesis that robot social intelligence is positively correlated with human compliance in an incidental human-robot encounter context. The video based dataset was also developed to obtain empirical evidence that can be used to design future real-world HRI studies. ## Dataset Contents * Four videos, each corresponding to a study condition. * Four sets of Perceived Social Intelligence Scale data. Each set corresponds to one study condition * Four sets of compliance likelihood questions, each set include one Likert question and one free-form question * One set of Godspeed questionnaire data. * One set of Anthropomorphism questionnaire data. * A csv file containing the participants demographic data, Likert scale data, and text responses. * A data dictionary explaining the meaning of each of the fields in the csv file. **More details and access to this data are available from the Texas Data Repository. ** This research is part of the Vid2Real project, supported by NSF Award #2219236 and Good Systems, a Grand Research Challenge at the University of Texas at Austin.

The Fields2Benhmark dataset is a collection of 350 agricultural fields in vector format manually selected to test agricultural coverage path planning algorithms. The files in this dataset are organized into two folders: - imgs/: contains the satellite images of the fields. - wkt/: has the vector data of the fields. A file with the same name in both folders corresponds to the same field. The first two letters in the name of the field indicate the country the field belongs to. The fields have been extracted from the EuroCrops dataset --specifically from the Netherlands, Estonia and Lithuania --, transformed each field into its own file, and converted them to Well-Known text (wkt). The satellite images were also provided, and they were created using the Mapping Toolbox from Matlab. ## Funding This dataset is provided as part of the project "Fields2Cover: Robust and efficient coverage paths for autonomous agricultural vehicles" (with project number ENPPS.LIFT.019.019 of the research programme Science PPP Fund for the top sectors which is (partly) financed by the Dutch Research Council (NWO). @article{Mier_Fields2Cover_An_open-source_2023, author={Mier, Gonzalo and Valente, João and de Bruin, Sytze}, journal={IEEE Robotics and Automation Letters}, title={Fields2Cover: An Open-Source Coverage Path Planning Library for Unmanned Agricultural Vehicles}, year={2023}, volume={8}, number={4}, pages={2166-2172}, doi={10.1109/LRA.2023.3248439} }

Towards automated analysis of large environments, hyperspectral sensors must be adapted into a format where they can be operated from mobile robots. In this dataset, we highlight hyperspectral datacubes collected from the Hyper-Drive imaging system. Our system collects and registers datacubes spanning the visible to shortwave infrared (660-1700 nm) in 33 wavelength channels. The system also simultaneously captures the ambient solar spectrum reflected off a white reference tile. The dataset consists of 500 labeled datacubes from on-road and off-road terrain compliant with the ATLAS. This work first appeared at WHISPERS 2023 In Athens, Greece and is a product of the Robotics and Intelligent Vehicles Research Laboratory (RIVeR) at Northeastern University. In addition to the 500 labeled hyperspectral datacubes, raw ROS bagfiles generated of each of the sensor feeds at a higher frame rate are available here. These files are provided as an additional resource and do not contain semantic labels, but contain ~10,000 additional hyperspectral datacubes of in-between frames from the labeled dataset. It also contains additional datatypes for terrain analysis such as inertial measurement unit (IMU) data. To the best of the authors knowledge, it is the largest vehicle-centric hyperspectral dataset currently available! This research was sponsored by the United States Army Core of Engineers (USACE) Engineer Research and Development Center (ERDC) Geospatial Research Laboratory (GRL) and was accomplished under Cooperative Agreement Federal Award Identification Number (FAIN) W9132V-22-2-0001. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of USACE EDRC GRL or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

Overview The goal: using simulation data to train neural networks to estimate the pose of a rover's camera with respect to a known target object The mission context: A simulated lunar surface, with lunar landers and lunar rovers. To accomplish their ressource extraction mission, the rovers must dig, transport and deliver regolith to a processing plant. For each of these tasks, a central need is for rovers to accurately estimate the relative pose both between themselves and with the landers. <img src="https://github.com/TeamL3/learned-pose-estimation/raw/main/misc/images/overview_low.png" alt="lunar surface overview" style="width:720px;"/> The dataset: - tf.data dataset ready for training (RGBD images and ground truth pose labels) - five different scenarios for relative pose estimation, some easier, some harder! <img src="https://github.com/TeamL3/learned-pose-estimation/raw/main/misc/images/dataset.png" alt="dataset samples" style="width:720px;"/> The code: Utilities for manipulating the dataset and calculating training metrics + example jupyter notebooks for data exploration and model training + more details on the dataset are available on github