Sufficient Exploration for Convex Q-learning
원문 초록
In recent years there has been a collective research effort to find new formulations of reinforcement learning that are simultaneously more efficient and more amenable to analysis. This paper concerns one approach that builds on the linear programming (LP) formulation of optimal control of Manne. A primal version is called logistic Q-learning, and a dual variant is convex Q-learning. This paper focuses on the latter, while building bridges with the former. The main contributions follow: (i) The dual of convex Q-learning is not precisely Manne's LP or a version of logistic Q-learning, but has similar structure that reveals the need for regularization to avoid over-fitting. (ii) A sufficient condition is obtained for a bounded solution to the Q-learning LP. (iii) Simulation studies reveal numerical challenges when addressing sampled-data systems based on a continuous time model. The challenge is addressed using state-dependent sampling. The theory is illustrated with applications to examples from OpenAI gym. It is shown that convex Q-learning is successful in cases where standard Q-learning diverges, such as the LQR problem.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Double Successive Over-Relaxation Q-Learning with an Extension to Deep Reinforcement Learning
Asymptotic Analysis of Sample-averaged Q-learning
Investigating the Performance and Reliability, of the Q-Learning Algorithm in Various Unknown Environments
Q-Learning under Finite Model Uncertainty
Personalized Exercise Recommendation with Semantically-Grounded Knowledge Tracing
ADDQ: Adaptive Distributional Double Q-Learning
토론
이 논문과 연결된 포럼 글이 아직 없습니다.