Referring to Screen Texts with Voice Assistants
원문 초록
Voice assistants help users make phone calls, send messages, create events, navigate, and do a lot more. However, assistants have limited capacity to understand their users' context. In this work, we aim to take a step in this direction. Our work dives into a new experience for users to refer to phone numbers, addresses, email addresses, URLs, and dates on their phone screens. Our focus lies in reference understanding, which becomes particularly interesting when multiple similar texts are present on screen, similar to visual grounding. We collect a dataset and propose a lightweight general-purpose model for this novel experience. Due to the high cost of consuming pixels directly, our system is designed to rely on the extracted text from the UI. Our model is modular, thus offering flexibility, improved interpretability, and efficient runtime memory utilization.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
GTA1: GUI Test-time Scaling Agent
High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learning
STRUCTSENSE: A Task-Agnostic Agentic Framework for Structured Information Extraction with Human-In-The-Loop Evaluation and Benchmarking
AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench
DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World
DrishtiKon: Multi-Granular Visual Grounding for Text-Rich Document Images
토론
이 논문과 연결된 포럼 글이 아직 없습니다.