Efficient Exploration

A computationally efficient approach for solving hard nonlinear problems of reinforcement learning (RL). It combines umbrella sampling, from computational physics/chemistry, with optimal control methods. The approach is realized on the basis of neural networks, with the use of policy gradient. It outperforms, by computational efficiency and implementation universality, the available state-of-the-art algorithms, in application to hard RL problems with sparse reward, state traps and lack of terminal states. The proposed approach uses an ensemble of simultaneously acting agents, with a modified reward which includes the ensemble entropy, yielding an optimal exploration-exploitation balance.

A Noisy Linear Layer is a linear layer with parametric noise added to the weights. This induced stochasticity can be used in reinforcement learning networks for the agent's policy to aid efficient exploration. The parameters of the noise are learned with gradient descent along with any other remaining network weights. Factorized Gaussian noise is the type of noise usually employed. The noisy linear layer takes the form: $$y = \left(b + Wx\right) + \left(b\_{noisy}\odot\epsilon^{b}+\left(W\_{noisy}\odot\epsilon^{w}\right)x\right) $$ where $\epsilon^{b}$ and $\epsilon^{w}$ are random variables.

NoisyNet-DQN is a modification of a DQN that utilises noisy linear layers for exploration instead of $\epsilon$-greedy exploration as in the original DQN formulation.

NoisyNet-A3C is a modification of A3C that utilises noisy linear layers for exploration instead of $\epsilon$-greedy exploration as in the original DQN formulation.

NoisyNet-Dueling is a modification of a Dueling Network that utilises noisy linear layers for exploration instead of $\epsilon$-greedy exploration as in the original Dueling formulation.