Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perception
원문 초록
Vision-language models (VLMs) have shown powerful capabilities in visual question answering and reasoning tasks by combining visual representations with the abstract skill set large language models (LLMs) learn during pretraining. Vision, while the most popular modality to augment LLMs with, is only one representation of a scene. In human-robot interaction scenarios, robot perception requires accurate scene understanding by the robot. In this paper, we define and demonstrate a method of aligning the embedding spaces of different modalities (in this case, inertial measurement unit (IMU) data) to the vision embedding space through a combination of supervised and contrastive training, enabling the VLM to understand and reason about these additional modalities without retraining. We opt to give the model IMU embeddings directly over using a separate human activity recognition model that feeds directly into the prompt to allow for any nonlinear interactions between the query, image, and IMU signal that would be lost by mapping the IMU data to a discrete activity label. Further, we demonstrate our methodology's efficacy through experiments involving human activity recognition using IMU data and visual inputs. Our results show that using multiple modalities as input improves the VLM's scene understanding and enhances its overall performance in various tasks, thus paving the way for more versatile and capable language models in multi-modal contexts.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations
PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assembly
Learning Sparsity for Effective and Efficient Music Performance Question Answering
Describe Anything Model for Visual Question Answering on Text-rich Images
Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Models
DrishtiKon: Multi-Granular Visual Grounding for Text-Rich Document Images
토론
이 논문과 연결된 포럼 글이 아직 없습니다.