Evaluations at Work: Measuring the Capabilities of GenAI in Use
원문 초록
Current AI benchmarks miss the messy, multi-turn nature of human-AI collaboration. We present an evaluation framework that decomposes real-world tasks into interdependent subtasks, letting us track both LLM performance and users' strategies across a dialogue. Complementing this framework, we develop a suite of metrics, including a composite usage derived from semantic similarity, word overlap, and numerical matches; structural coherence; intra-turn diversity; and a novel measure of the "information frontier" reflecting the alignment between AI outputs and users' working knowledge. We demonstrate our methodology in a financial valuation task that mirrors real-world complexity. Our empirical findings reveal that while greater integration of LLM-generated content generally enhances output quality, its benefits are moderated by factors such as response incoherence, excessive subtask diversity, and the distance of provided information from users' existing knowledge. These results suggest that proactive dialogue strategies designed to inject novelty may inadvertently undermine task performance. Our work thus advances a more holistic evaluation of human-AI collaboration, offering both a robust methodological framework and actionable insights for developing more effective AI-augmented work processes.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Models
FA: Forced Prompt Learning of Vision-Language Models for Out-of-Distribution Detection
ImpliRet: Benchmarking the Implicit Fact Retrieval Challenge
Trend-Aware Fashion Recommendation with Visual Segmentation and Semantic Similarity
KNN-Defense: Defense against 3D Adversarial Point Clouds using Nearest-Neighbor Search
IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory
토론
이 논문과 연결된 포럼 글이 아직 없습니다.