Learning Policies from Self-Play with Policy Gradients and MCTS Value Estimates
원문 초록
In recent years, state-of-the-art game-playing agents often involve policies that are trained in self-playing processes where Monte Carlo tree search (MCTS) algorithms and trained policies iteratively improve each other. The strongest results have been obtained when policies are trained to mimic the search behaviour of MCTS by minimising a cross-entropy loss. Because MCTS, by design, includes an element of exploration, policies trained in this manner are also likely to exhibit a similar extent of exploration. In this paper, we are interested in learning policies for a project with future goals including the extraction of interpretable strategies, rather than state-of-the-art game-playing performance. For these goals, we argue that such an extent of exploration is undesirable, and we propose a novel objective function for training policies that are not exploratory. We derive a policy gradient expression for maximising this objective function, which can be estimated using MCTS value estimates, rather than MCTS visit counts. We empirically evaluate various properties of resulting policies, in a variety of board games.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Adaptive Action Duration with Contextual Bandits for Deep Reinforcement Learning in Dynamic Environments
Vision Language Models are Biased
4Hammer: a board-game reinforcement learning environment for the hour long time frame
Playing Non-Embedded Card-Based Games with Reinforcement Learning
ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition
Playing Hex and Counter Wargames using Reinforcement Learning and Recurrent Neural Networks
토론
이 논문과 연결된 포럼 글이 아직 없습니다.