Beyond Third-Person Audits: Situated Interaction Auditing for User-Centered LLM Bias Research

원문 초록

Research on bias in large language models (LLMs) has predominantly focused on third-person audits, which study how models represent or evaluate demographic groups as external subjects. However, this paradigm overlooks a structural blind spot because the user is absent from the audit. In practice, LLMs are used in open-ended, personal interactions, during which the model implicitly represents the user and adjusts its responses accordingly. When identical requests yield different responses depending on who is asking, bias manifests not in how the model describes others but in how it treats its interlocutor. Empirical first-person bias studies are multiplying, but they use different cues, outcomes, and validation choices, which makes their results hard to compare. We propose Situated Interaction Auditing (SIA), a shared diagnostic framework for this literature. It introduces a typology of user profile signals---names and surface markers, writing style and dialect, and stated identity---together with design principles and outcome measures that let studies be compared and accumulated. We instantiate the framework in a case study of Chilean first and last names across nine task domains and three models, and we close with guidance for practitioners and open questions for researchers.