CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Data
원문 초록
Advances in generative AI point towards a new era of personalized applications that perform diverse tasks on behalf of users. While general AI assistants have yet to fully emerge, their potential to share personal data raises significant privacy challenges. This paper introduces CI-Bench, a comprehensive synthetic benchmark for evaluating the ability of AI assistants to protect personal information during model inference. Leveraging the Contextual Integrity framework, our benchmark enables systematic assessment of information flow across important context dimensions, including roles, information types, and transmission principles. We present a novel, scalable, multi-step synthetic data pipeline for generating natural communications, including dialogues and emails. Unlike previous work with smaller, narrowly focused evaluations, we present a novel, scalable, multi-step data pipeline that synthetically generates natural communications, including dialogues and emails, which we use to generate 44 thousand test samples across eight domains. Additionally, we formulate and evaluate a naive AI assistant to demonstrate the need for further study and careful training towards personal assistant tasks. We envision CI-Bench as a valuable tool for guiding future language model development, deployment, system design, and dataset construction, ultimately contributing to the development of AI assistants that align with users' privacy expectations.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
mTSBench: Benchmarking Multivariate Time Series Anomaly Detection and Model Selection at Scale
BMFM-RNA: An Open Framework for Building and Evaluating Transcriptomic Foundation Models
MLAR: Multi-layer Large Language Model-based Robotic Process Automation Applicant Tracking
A Systematic Analysis of Hybrid Linear Attention
Inaugural MOASEI Competition at AAMAS'2025: A Technical Report
Finance Language Model Evaluation (FLaME)
토론
이 논문과 연결된 포럼 글이 아직 없습니다.