A Unified Taxonomy and Evaluation Framework for Distributional Shifts in Reinforcement Learning

원문 초록

Distributional shift in reinforcement learning (RL) is defined inconsistently across the literature, and it is studied alongside out-of-distribution (OOD) generalization and non-stationarity without a clear account of how the three terms are related. Methods are also often described as robust after evaluation under only a few particular changes, leaving the scope of such claims unclear. We develop a unified taxonomy that brings the three terms under one definition of distributional shift. We factorize the joint distribution of a one-step interaction into five components, each produced by its own generating function: initial-state distribution, observation mechanism, policy implementation (distinct from the policy being optimized), transition dynamics, and reward mechanism. A shift is specified by what changes (the changed components) and when it changes (the interaction regime: OOD, non-stationary, or combined), while changes in interaction data and performance are treated as observed effects. We complement the taxonomy with an evaluation framework that measures immediate generalization and subsequent adaptation, summarizing adaptation in a shift adaptation score (SAS). Published benchmark results, controlled Gridworld experiments with DQN, A2C, and PPO, and a mapping of selected studies show that method responses and rankings vary with the shifted component, and that existing evaluations cover only some components. Robustness claims should therefore be bounded by the components and regimes evaluated, since strong performance under one shift does not establish robustness under others. Identifying an unknown component-level source from observed interaction data remains a direction for future research.