Cancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processes
원문 초록
Constrained Markov Decision Processes (CMDPs) are one of the common ways to model safe reinforcement learning problems, where constraint functions model the safety objectives. Lagrangian-based dual or primal-dual algorithms provide efficient methods for learning in CMDPs. For these algorithms, the currently known regret bounds in the finite-horizon setting allow for a "cancellation of errors"; one can compensate for a constraint violation in one episode with a strict constraint satisfaction in another. However, we do not consider such a behavior safe in practical applications. In this paper, we overcome this weakness by proposing a novel model-based dual algorithm OptAug-CMDP for tabular finite-horizon CMDPs. Our algorithm is motivated by the augmented Lagrangian method and can be performed efficiently. We show that during $K$ episodes of exploring the CMDP, our algorithm obtains a regret of $\tilde{O}(\sqrt{K})$ for both the objective and the constraint violation. Unlike existing Lagrangian approaches, our algorithm achieves this regret without the need for the cancellation of errors.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Reinforcement Learning from Human Feedback with High-Confidence Safety Constraints
Safe Multi-Agent Navigation guided by Goal-Conditioned Safe Reinforcement Learning
Risk-Averse Reinforcement Learning: An Optimal Transport Perspective on Temporal Difference Learning
Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning
Offline Safe Reinforcement Learning Using Trajectory Classification
Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning
토론
이 논문과 연결된 포럼 글이 아직 없습니다.