Contrast Sets for Evaluating Language-Guided Robot Policies
원문 초록
Robot evaluations in language-guided, real world settings are time-consuming and often sample only a small space of potential instructions across complex scenes. In this work, we introduce contrast sets for robotics as an approach to make small, but specific, perturbations to otherwise independent, identically distributed (i.i.d.) test instances. We investigate the relationship between experimenter effort to carry out an evaluation and the resulting estimated test performance as well as the insights that can be drawn from performance on perturbed instances. We use the relative performance change of different contrast set perturbations to characterize policies at reduced experimenter effort in both a simulated manipulation task and a physical robot vision-and-language navigation task. We encourage the use of contrast set evaluations as a more informative alternative to small scale, i.i.d. demonstrations on physical robots, and as a scalable alternative to industry-scale real world evaluations.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments
A Navigation Framework Utilizing Vision-Language Models
Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigation
FlightGPT: Towards Generalizable and Interpretable UAV Vision-and-Language Navigation with Vision-Language Models
Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation
CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memory
토론
이 논문과 연결된 포럼 글이 아직 없습니다.