Parallel $Q$-Learning: Scaling Off-policy Reinforcement Learning under Massively Parallel Simulation
원문 초록
Reinforcement learning is time-consuming for complex tasks due to the need for large amounts of training data. Recent advances in GPU-based simulation, such as Isaac Gym, have sped up data collection thousands of times on a commodity GPU. Most prior works used on-policy methods like PPO due to their simplicity and ease of scaling. Off-policy methods are more data efficient but challenging to scale, resulting in a longer wall-clock training time. This paper presents a Parallel $Q$-Learning (PQL) scheme that outperforms PPO in wall-clock time while maintaining superior sample efficiency of off-policy learning. PQL achieves this by parallelizing data collection, policy learning, and value learning. Different from prior works on distributed off-policy learning, such as Apex, our scheme is designed specifically for massively parallel GPU-based simulation and optimized to work on a single workstation. In experiments, we demonstrate that $Q$-learning can be scaled to \textit{tens of thousands of parallel environments} and investigate important factors affecting learning speed. The code is available at https://github.com/Improbable-AI/pql.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
AutoTriton: Automatic Triton Programming with Reinforcement Learning in LLMs
CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learning
Artificial Generals Intelligence: Mastering Generals.io with Reinforcement Learning
Reinforcement Learning-Based Policy Optimisation For Heterogeneous Radio Access
ProteinZero: Self-Improving Protein Generation via Online Reinforcement Learning
Investigating the Performance and Reliability, of the Q-Learning Algorithm in Various Unknown Environments
토론
이 논문과 연결된 포럼 글이 아직 없습니다.