Efficient exploration remains a major challenge for reinforcement learning. One reason is that the variability of the returns often depends on the current state and action, and is therefore heteroscedastic. Classical exploration strategies such as upper confidence bound algorithms and Thompson sampling fail to appropriately account for heteroscedasticity, even in the bandit setting. Motivated by recent findings that address this issue in bandits, we propose to use Information-Directed Sampling (IDS) for exploration in reinforcement learning. As our main contribution, we build on recent advances in distributional reinforcement learning and propose a novel, tractable approximation of IDS for deep Q-learning. The resulting exploration strategy explicitly accounts for both parametric uncertainty and heteroscedastic observation noise. We evaluate our method on Atari games and demonstrate a significant improvement over alternative approaches.
In this paper, we propose the Quantile Option Architecture (QUOTA) for exploration based on recent advances in distributional reinforcement learning (RL). In QUOTA, decision making is based on quantiles of a value distribution, not only the mean. QUOTA provides a new dimension for exploration via making use of both optimism and pessimism of a value distribution. We demonstrate the performance advantage of QUOTA in both challenging video games and physical robot simulators.
In this work, we build on recent advances in distributional reinforcement learning to give a generally applicable, flexible, and state-of-the-art distributional variant of DQN. We achieve this by using quantile regression to approximate the full quantile function for the state-action return distribution. By reparameterizing a distribution over the sample space, this yields an implicitly defined return distribution and gives rise to a large class of risk-sensitive policies. We demonstrate improved performance on the 57 Atari 2600 games in the ALE, and use our algorithm's implicitly defined distributions to study the effects of risk-sensitive policies in Atari games.
In this work we present a new agent architecture, called Reactor, which combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN (Wang et al., 2016) and Categorical DQN (Bellemare et al., 2017), while giving better run-time performance than A3C (Mnih et al., 2016). Our first contribution is a new policy evaluation algorithm called Distributional Retrace, which brings multi-step off-policy updates to the distributional reinforcement learning setting. The same approach can be used to convert several classes of multi-step policy evaluation algorithms designed for expected value evaluation into distributional ones. Next, we introduce the \b{eta}-leave-one-out policy gradient algorithm which improves the trade-off between variance and bias by using action values as a baseline. Our final algorithmic contribution is a new prioritized replay algorithm for sequences, which exploits the temporal locality of neighboring observations for more efficient replay prioritization. Using the Atari 2600 benchmarks, we show that each of these innovations contribute to both the sample efficiency and final agent performance. Finally, we demonstrate that Reactor reaches state-of-the-art performance after 200 million frames and less than a day of training.
In reinforcement learning an agent interacts with the environment by taking actions and observing the next state and reward. When sampled probabilistically, these state transitions, rewards, and actions can all induce randomness in the observed long-term return. Traditionally, reinforcement learning algorithms average over this randomness to estimate the value function. In this paper, we build on recent work advocating a distributional approach to reinforcement learning in which the distribution over returns is modeled explicitly instead of only estimating the mean. That is, we examine methods of learning the value distribution instead of the value function. We give results that close a number of gaps between the theoretical and algorithmic results given by Bellemare, Dabney, and Munos (2017). First, we extend existing results to the approximate distribution setting. Second, we present a novel distributional reinforcement learning algorithm consistent with our theoretical formulation. Finally, we evaluate this new algorithm on the Atari 2600 games, observing that it significantly outperforms many of the recent improvements on DQN, including the related distributional algorithm C51.
Distributional approaches to value-based reinforcement learning model the entire distribution of returns, rather than just their expected values, and have recently been shown to yield state-of-the-art empirical performance. This was demonstrated by the recently proposed C51 algorithm, based on categorical distributional reinforcement learning (CDRL) [Bellemare et al., 2017]. However, the theoretical properties of CDRL algorithms are not yet well understood. In this paper, we introduce a framework to analyse CDRL algorithms, establish the importance of the projected distributional Bellman operator in distributional RL, draw fundamental connections between CDRL and the Cram\'er distance, and give a proof of convergence for sample-based categorical distributional reinforcement learning algorithms.
We propose a framework based on distributional reinforcement learning and recent attempts to combine Bayesian parameter updates with deep reinforcement learning. We show that our proposed framework conceptually unifies multiple previous methods in exploration. We also derive a practical algorithm that achieves efficient exploration on challenging control tasks.
We devise a distributional variant of gradient temporal-difference (TD) learning. Distributional reinforcement learning has been demonstrated to outperform the regular one in the recent study \citep{bellemare2017distributional}. In the policy evaluation setting, we design two new algorithms called distributional GTD2 and distributional TDC using the Cram{\'e}r distance on the distributional version of the Bellman error objective function, which inherits advantages of both the nonlinear gradient TD algorithms and the distributional RL approach. In the control setting, we propose the distributional Greedy-GQ using the similar derivation. We prove the asymptotic almost-sure convergence of distributional GTD2 and TDC to a local optimal solution for general smooth function approximators, which includes neural networks that have been widely used in recent study to solve the real-life RL problems. In each step, the computational complexities of above three algorithms are linear w.r.t.\ the number of the parameters of the function approximator, thus can be implemented efficiently for neural networks.
Distributional reinforcement learning (distributional RL) has seen empirical success in complex Markov Decision Processes (MDPs) in the setting of nonlinear function approximation. However, there are many different ways in which one can leverage the distributional approach to reinforcement learning. In this paper, we propose GAN Q-learning, a novel distributional RL method based on generative adversarial networks (GANs) and analyze its performance in simple tabular environments, as well as OpenAI Gym. We empirically show that our algorithm leverages the flexibility and blackbox approach of deep learning models while providing a viable alternative to traditional methods.
본 논문에서는 크라메르(Cramér) 기하에서의 스칼라 범주형 TD(CTD)와 최대 평균 불일치(MMD) 기하에서의 다변량 부호 범주형 TD(MTD)를 포괄하여, 비동기 범주형 분포 시차 학습(distributional temporal-difference) 방법의 유한 반복(finite-iteration) 거동을 연구한다. 저자들은 i.i.d. 샘플링 및 연속적인 마르코프 궤적 하에서 높은 확률의 이론적 보장을 확립한다. CTD의 경우, 분산 감소나 데이터 삭제 기법을 사용하지 않고도 실제 반환 분포에 도달하기 위한 크라메르 표본 복잡도의 주요 항이 i.i.d. 설정에서는 $\tilde O(ρ_{\min}^{-1}(1-γ)^{-2}\varepsilon^{-2})$, 마르코프 설정에서는 $\tilde O(μ_{\min}^{-1}(1-γ)^{-2}\varepsilon^{-2}+μ_{\min}^{-1}t_{\mathrm{mix}})$임을 보인다. MTD의 경우, MMD 표본 복잡도의 주요 항은 $\tilde O(ρ_{\min}^{-1}(1-γ)^{-1-c}\varepsilon^{-2})$ 및 $\tilde O(μ_{\min}^{-1}(1-γ)^{-1-c}\varepsilon^{-2}+μ_{\min}^{-1}t_{\mathrm{mix}})$이며(여기서 $c\in(0,2)$는 MMD 커널의 동차 지수), 이는 $c=1$일 때 CTD 수렴 속도로 단순화된다. 이는 저자들이 알기로 MTD에 대한 최초의 유한 반복 수렴 속도이다. 무할인 고정 지평(undiscounted fixed-horizon) 정책 평가의 경우, 동일한 분석이 i.i.d. 및 에피소딕 샘플링 하의 CTD와 MTD 고정 지평 버전에 적용되며, 유효 지평 $(1-γ)^{-1}$이 주요 항에서 지평 $H$로 대체되어 할인 설정의 수렴 속도와 일치한다. 일치하는 미니맥스(minimax) 하한은 주요 항과 내포된 1-바서슈타인(Wasserstein) 속도가 i.i.d., 마르코프, 에피소딕 샘플링 하에서 최적임을 보여준다. 종합적으로 이러한 결과들은 스칼라, 다변량, 할인 및 고정 지평 설정 전반에 걸쳐 비동기 범주형 분포 TD에 대한 통합된 비점근적(non-asymptotic) 분석을 제공하며, 주요 항에서 미니맥스 최적의 수렴 속도를 달성한다.
이 논문에서는 통계적 효율성의 관점에서 분위수 기반 분포적 강화학습(distributional reinforcement learning)을 연구한다. 우리는 분포적 정책 평가(distributional policy evaluation)에 초점을 맞추며, 그 목표는 주어진 정책 하에서 할인된 누적 보상의 분포, 즉 리턴 분포(return distribution)를 특성화하는 것이다. 리턴 분포의 유한 차원 표현을 얻기 위해, 분위수 투영 분포적 벨만 방정식(quantile-projected distributional Bellman equation)에 의해 유도된 분위수 고정점(quantile fixed point) $η_m$을 고려한다. 생성 모델에 대한 접근을 가정하여, 경험적 마르코프 결정 과정을 기반으로 추정량 $η_m^{(n)}$을 구성한다. 고정된 분위수 수 $m$에 대해, 상한 $W_\infty$ 거리 하에서 $η_m^{(n)}$과 $η_m$ 사이의 비점근적 오차 경계를 확립하며, 추정 오차가 $m$과 $n$에 대해 $\widetilde{O}(\sqrt{m/n})$으로 스케일됨을 보인다. 이는 분위수 기반 분포적 정책 평가 문제가 표본 효율적으로 해결될 수 있으며, 최적 모수적 $\sqrt{n}$ 수렴률을 달성함을 의미한다. 우리는 분위수 파라미터 $\sqrt{n}(θ_m^{(n)}-θ_m)$의 점근 분포를 유도하고 반모수 효율성 경계(semiparametric efficiency bound)를 특성화하며, 이 경계는 우리의 추정량에 의해 달성된다. 고정 차원 설정을 넘어, 분위수 수가 발산하는 점근적 체제를 조사한다. 극한 공분산 구조를 특성화하고, 이것이 분포적 정책 평가를 위한 비모수 모델의 반모수 효율성 경계와 일치함을 보임으로써, 분위수 기반 추정량이 무한 차원 극한에서도 점근적으로 효율적임을 보인다. 마지막으로, 매끄러운 함수형(smooth functionals) $\sqrt{n}(η_m^{(n)}(s)-η_m(s))f$에 대한 Berry-Esseen 정리를 확립함으로써, 분위수 투영된 리턴 분포의 함수형에 대한 통계적으로 유효한 추론의 기반을 제공한다.