UDQL: Bridging The Gap between MSE Loss and The Optimal Value Function in Offline Reinforcement Learning
원문 초록
The Mean Square Error (MSE) is commonly utilized to estimate the solution of the optimal value function in the vast majority of offline reinforcement learning (RL) models and has achieved outstanding performance. However, we find that its principle can lead to overestimation phenomenon for the value function. In this paper, we first theoretically analyze overestimation phenomenon led by MSE and provide the theoretical upper bound of the overestimated error. Furthermore, to address it, we propose a novel Bellman underestimated operator to counteract overestimation phenomenon and then prove its contraction characteristics. At last, we propose the offline RL algorithm based on underestimated operator and diffusion policy model. Extensive experimental results on D4RL tasks show that our method can outperform state-of-the-art offline RL algorithms, which demonstrates that our theoretical analysis and underestimation way are effective for offline RL tasks.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood
From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learning
Policy-Based Trajectory Clustering in Offline Reinforcement Learning
Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity
CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimization
DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty
토론
이 논문과 연결된 포럼 글이 아직 없습니다.