Embodied Question Answering

The EQA (Embodied Question Answering) dataset is a dataset of visual questions and answers grounded in House3D. For this dataset an agent is spawned at a random location in a 3D environment and asked a question (for e.g. "What color is the car?"). In order to answer, the agent must first intelligently navigate to explore the environment, gather necessary visual information through first-person vision, and then answer the question ("orange").

The OpenEQA dataset is a significant contribution in the field of Embodied Question Answering (EQA). Let me provide you with some details: 1. Definition: - Embodied Question Answering (EQA) involves understanding an environment well enough to answer questions about it in natural language. - EQA agents can achieve this understanding through either episodic memory (as seen in agents using smart glasses) or active exploration of the environment (as in the case of mobile robots). 2. OpenEQA Dataset: - OpenEQA is the first open-vocabulary benchmark dataset for EQA that supports both episodic memory and active exploration use cases. - It contains over 1600 high-quality human-generated questions drawn from more than 180 real-world environments. - The dataset consists of question-answer pairs ($Q, A^*$) and corresponding episode histories ($H$). - You can find the question-answer pairs in the file data/open-eqa-v0.json. - To access the episode histories, follow the instructions provided here. 3. Evaluation Protocol: - OpenEQA also provides an automatic evaluation protocol powered by language model-based evaluation (LLM). - This evaluation protocol correlates well with human judgment. 4. Foundation Models Evaluation: - Researchers evaluated several state-of-the-art foundation models, including GPT-4V, using the OpenEQA dataset. - The findings revealed that these models significantly lag behind human-level performance in EQA tasks. 5. Significance: - OpenEQA serves as a straightforward, measurable, and practically relevant benchmark for current-generation foundation models. - It poses a considerable challenge and inspires research at the intersection of Embodied AI, conversational agents, and world models.