Anchors in the Machine: Behavioral and Attributional Evidence of Anchoring Bias in LLMs

원문 초록

Large language models (LLMs) are increasingly studied as behavioral subjects (e.g., synthetic participants or users) and decision systems (e.g., LLM-as-a-judge), yet it remains unclear how observed cognitive biases relate to changes in model output probabilities. Anchoring bias, a classic human judgment bias, provides a useful test case. Prior research largely relies on generated responses, leaving distributional effects and input-field contributions less explored. This paper advances the study of anchoring in LLMs through three contributions: (1) a log-probability-based behavioral analysis examining how numerical anchors shift distributions over candidate outputs, using alternative questions and anchor pairs alongside a positive control to reduce reliance on potentially memorized stimuli; (2) exact Banzhaf-value attribution over structured prompt fields to quantify anchor contributions to log-probabilities; and (3) an Anchoring Bias Sensitivity Score (ABSS) integrating behavioral and attributional evidence across six open-source models. Results reveal predominantly positive behavioral anchoring effects, varying substantially across models. Gemma-2B, Phi-2, and Llama-2-7B achieve the highest ABSS rankings, whereas other models exhibit weaker behavioral-attributional alignment despite sometimes showing substantial behavioral shifts. Attributional effects also vary across prompts, indicating that behavioral anchoring does not necessarily imply stronger anchor-field contributions. These findings raise questions about LLM reliability as synthetic participants and potential vulnerabilities to numerical cues in automated evaluation and decision-making. More broadly, the framework connects behavioral science with explainable AI, particularly interpretability, and AI safety, providing a reproducible approach to investigating cognitive-bias-like effects in LLMs.