Vision and Language Navigation

Room-Across-Room (RxR) is a multilingual dataset for Vision-and-Language Navigation (VLN) for Matterport3D environments. In contrast to related datasets such as Room-to-Room (R2R), RxR is 10x larger, multilingual (English, Hindi and Telugu), with longer and more variable paths, and it includes and fine-grained visual groundings that relate each word to pixels/surfaces in the environment. Source: Room-Across-Room (RxR) Dataset

An interactive, first-person, partially-observed visual environment that uses Google Street View for its photographic content and broad coverage, and give performance baselines for a challenging goal-driven navigation task. Source: The StreetLearn Environment and Dataset

MINOS is a simulator designed to support the development of multisensory models for goal-directed navigation in complex indoor environments. MINOS leverages large datasets of complex 3D environments and supports flexible configuration of multimodal sensor suites.

Touchdown is a corpus for executing navigation instructions and resolving spatial descriptions in visual real-world environments. The task is to follow instruction to a goal position and there find a hidden object, Touchdown the bear. Source: https://github.com/lil-lab/touchdown Image Source: https://github.com/lil-lab/touchdown

Talk The Walk is a large-scale dialogue dataset grounded in action and perception. The task involves two agents (a “guide” and a “tourist”) that communicate via natural language in order to achieve a common goal: having the tourist navigate to a given target location.

CHALET is a 3D house simulator with support for navigation and manipulation. Unlike existing systems, CHALET supports both a wide range of object manipulation, as well as supporting complex environemnt layouts consisting of multiple rooms. The range of object manipulations includes the ability to pick up and place objects, toggle the state of objects like taps or televesions, open or close containers, and insert or remove objects from these containers. In addition, the simulator comes with 58 rooms that can be combined to create houses, including 10 default house layouts. CHALET is therefore suitable for setting up challenging environments for various AI tasks that require complex language understanding and planning, such as navigation, manipulation, instruction following, and interactive question answering. Source: https://github.com/lil-lab/chalet Image Source: https://arxiv.org/pdf/1809.00786.pdf

WebLINX is a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. It covers a broad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios.

7,672 human written natural language navigation instructions for routes in OpenStreetMap with a focus on visual landmarks. Validated in Street View.

The RUN dataset is based on OpenStreetMap (OSM). The map contains rich layers and an abundance of entities of different types. Each entity is complex and can contain (at least) four labels: name, type, is building=y/n, and house number. An entity can spread over several tiles. As the maps do not overlap, only very few entities are shared among them. The RUN dataset aligns NL navigation instructions to coordinates of their corresponding route on the OSM map. Source: RUN

A dataset (in English; and also extended to Hindi) with human-written navigation and assembling instructions, and the corresponding ground truth trajectories. Source: ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments

This dataset enriches the benchmark Room-to-Room (R2R) dataset by dividing the instructions into sub-instructions and pairing each of those with their corresponding viewpoints in the path. The overall instruction and trajectory of each sample remains the same. Source: https://github.com/YicongHong/Fine-Grained-R2R

The Robo-VLN dataset is a continuous control formulation of the VLN-CE dataset by Krantz et al ported over from Room-to-Room (R2R) dataset created by Anderson et al. The details regarding converting discrete VLN dataset into continuous control formulation can be found in our paper. | Dataset | Path to extract | Size | |-------------- |---------------------------- |------- | | robo_vln_v1.zip | data/datasets/robo_vln_v1 | 76.9 MB | #### Robo-VLN Dataset The dataset robo_vln_v1 contains the train, val_seen, and val_unseen splits. * train: 7739 episodes * val_seen: 570 episodes * val_unseen: 1224 episodes Format of {split}.json.gz { 'episodes' = [ { 'episode_id': 4991, 'trajectory_id': 3279, 'scene_id': 'mp3d/JeFG25nYj2p/JeFG25nYj2p.glb', 'instruction': { 'instruction_text': 'Walk past the striped area rug...', 'instruction_tokens': [2384, 1589, 2202, 2118, 133, 1856, 9] }, 'start_position': [10.257800102233887, 0.09358400106430054, -2.379739999771118], 'start_rotation': [0, 0.3332950713608026, 0, 0.9428225683587541], 'goals': [ { 'position': [3.360340118408203, 0.09358400106430054, 3.07817006111145], 'radius': 3.0 } ], 'reference_path': [ [10.257800102233887, 0.09358400106430054, -2.379739999771118], [9.434900283813477, 0.09358400106430054, -1.3061100244522095] ... [3.360340118408203, 0.09358400106430054, 3.07817006111145], ], 'info': {'geodesic_distance': 9.65537166595459}, }, ... ], 'instruction_vocab': [ 'word_list': [..., 'orchids', 'order', 'orient', ...], 'word2idx_dict': { ..., 'orchids': 1505, 'order': 1506, 'orient': 1507, ... }, 'itos': [..., 'orchids', 'order', 'orient', ...], 'stoi': { ..., 'orchids': 1505, 'order': 1506, 'orient': 1507, ... }, 'num_vocab': 2504, 'UNK_INDEX': 1, 'PAD_INDEX': 0, ] } * Format of {split}_gt.json.gz { '4991': { 'actions': [ ... [-0.999969482421875, 1.0], [-0.9999847412109375, 0.15731772780418396], ... ], 'forward_steps': 325, 'locations': [ [10.257800102233887, 0.09358400106430054, -2.379739999771118], [10.257800102233887, 0.09358400106430054, -2.379739999771118], ... [-12.644463539123535, 0.1518409252166748, 4.2241311073303220] ] } ... }

Talk2Nav is a large-scale dataset with verbal navigation instructions. Source: Talk2Nav: Long-Range Vision-and-Language Navigation with Dual Attention and Spatial Memory Image Source: https://www.trace.ethz.ch/publications/2019/talk2nav/index.html