A Near-Optimal Primal-Dual Method for Off-Policy Learning in CMDP
원문 초록
As an important framework for safe Reinforcement Learning, the Constrained Markov Decision Process (CMDP) has been extensively studied in the recent literature. However, despite the rich results under various on-policy learning settings, there still lacks some essential understanding of the offline CMDP problems, in terms of both the algorithm design and the information theoretic sample complexity lower bound. In this paper, we focus on solving the CMDP problems where only offline data are available. By adopting the concept of the single-policy concentrability coefficient $C^*$, we establish an $\Omega\left(\frac{\min\left\{|\mathcal{S}||\mathcal{A}|,|\mathcal{S}|+I\right\} C^*}{(1-\gamma)^3\epsilon^2}\right)$ sample complexity lower bound for the offline CMDP problem, where $I$ stands for the number of constraints. By introducing a simple but novel deviation control mechanism, we propose a near-optimal primal-dual learning algorithm called DPDL. This algorithm provably guarantees zero constraint violation and its sample complexity matches the above lower bound except for an $\tilde{\mathcal{O}}((1-\gamma)^{-1})$ factor. Comprehensive discussion on how to deal with the unknown constant $C^*$ and the potential asynchronous structure on the offline dataset are also included.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Reinforcement Learning from Human Feedback with High-Confidence Safety Constraints
Safe Multi-Agent Navigation guided by Goal-Conditioned Safe Reinforcement Learning
Risk-Averse Reinforcement Learning: An Optimal Transport Perspective on Temporal Difference Learning
Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning
Offline Safe Reinforcement Learning Using Trajectory Classification
Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning
토론
이 논문과 연결된 포럼 글이 아직 없습니다.