You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
원문 초록
The goal of offline reinforcement learning (RL) is to find an optimal policy given prerecorded trajectories. Many current approaches customize existing off-policy RL algorithms, especially actor-critic algorithms in which policy evaluation and improvement are iterated. However, the convergence of such approaches is not guaranteed due to the use of complex non-linear function approximation and an intertwined optimization process. By contrast, we propose a simple baseline algorithm for offline RL that only performs the policy evaluation step once so that the algorithm does not require complex stabilization schemes. Since the proposed algorithm is not likely to converge to an optimal policy, it is an appropriate baseline for actor-critic algorithms that ought to be outperformed if there is indeed value in iterative optimization in the offline setting. Surprisingly, we empirically find that the proposed algorithm exhibits competitive and sometimes even state-of-the-art performance in a subset of the D4RL offline RL benchmark. This result suggests that future work is needed to fully exploit the potential advantages of iterative optimization in order to justify the reduced stability of such methods.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood
From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learning
Policy-Based Trajectory Clustering in Offline Reinforcement Learning
Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity
CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimization
DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty
토론
이 논문과 연결된 포럼 글이 아직 없습니다.