PromptShift-CRC: What Drift-Aware Weighting Adds to Online Conformal Risk Control for Foundation Models

원문 초록

A risk threshold calibrated on old prompts can stop being valid when prompts to a foundation model drift. We study PromptShift-CRC, a rule that weights past prompts by feature similarity, falls back to recent prompts when the current prompt is unlike the past, and updates its level online. We prove that the update controls the long-run violation rate for any weights, so marginal coverage cannot separate weighting schemes. We give a conditional bound at a fixed level whose cost is a weighted total variation distance, bounded by a computable prompt-drift term up to an unknown Lipschitz constant. We propose a comparison protocol: a matched short-window baseline, a validity and sharpness frontier, 30-seed synthetic streams, and real-output streams ordered without using the model's output. On MMLU, TruthfulQA (Qwen2.5-1.5B) and RealToxicityPrompts (GPT-2, replicated with a second generator and classifier), every online method reaches the target. PromptShift-CRC stays within 0.0055 of the Adaptive CRC frontier and avoids the worst window, but DtACI matches or beats it in most settings. The rule performs like a mid-ranked window without having to choose one. Tests fixed in advance to favor the weights show no consistent gain over the frontier. Code, outputs and streams are released.