Generative OOD-regularized Model-based Policy Optimization

원문 초록

We study sequential decision-making with offline reinforcement learning (RL). Traditional offline RL policies may result in out-of-distribution (OOD) actions when training relies only on sparse offline representations. To ensure safe offline policies in a sparse state-action space, we explore how density estimation models can be integrated into model-based RL methods to avoid OOD regions. Generative models are capable of explicitly modeling the density in sparse state-action spaces. Building on this, we introduce Generative OOD-regularized Model-based Policy Optimization (GORMPO), a density-regularized offline RL algorithm that uses generative density modeling to restrict policy updates to high-density areas of the dataset. Furthermore, we examine whether better OOD detection corresponds to better model-based offline policies. We compare (1) the OOD detection capabilities of various density estimators and (2) their performance within the GORMPO framework on a real-world medical dataset and sparse offline RL datasets. We provide theoretical guarantees bounding GORMPO's performance gap in terms of density estimation error, ensuring near-optimal policies under imperfect density estimates. Empirically, GORMPO remains competitive in reward and improves clinical metrics compared to state-of-the-art clinical RL models, and outperforms state-of-the-art offline RL models in offline RL benchmarks. Our empirical findings show that better OOD detection generally results in improved policies in environments with stable dynamics, while conservative penalties with poor density estimation are favored when dynamics are uncertain. Code, checkpoints, sparse, and OOD D4RL datasets are available at https://github.com/Rose-STL-Lab/GORMPO.