PyTorchKR Papers

최신 논문

총 445편
태그·기간 등 상세 필터

카테고리 탐색

로그인 후 이 태그 구독

arXiv:1802.07834공식 코드 · Papers with Code

Learning to Gather without Communication

A standard belief on emerging collective behavior is that it emerges from simple individual rules. Most of the mathematical research on such collective behavior starts from imperative individual rules, like always go to the center. But how could an (optimal) individual rule emerge during a short period within the group lifetime, especially if communication is not available. We argue that such rules can actually emerge in a group in a short span of time via collective (multi-agent) reinforcement learning, i.e learning via rewards and punishments. We consider the gathering problem: several agents (social animals, swarming robots...) must gather around a same position, which is not determined in advance. They must do so without communication on their planned decision, just by looking at the position of other agents. We present the first experimental evidence that a gathering behavior can be learned without communication in a partially observable environment. The learned behavior has the same properties as a self-stabilizing distributed algorithm, as processes can gather from any initial state (and thus tolerate any transient failure). Besides, we show that it is possible to tolerate the brutal loss of up to 90\% of agents without significant impact on the behavior.

arXiv:1709.04326공식 코드 · Papers with Code

Learning with Opponent-Learning Awareness

Multi-agent settings are quickly gathering importance in machine learning. This includes a plethora of recent work on deep multi-agent reinforcement learning, but also can be extended to hierarchical RL, generative adversarial networks and decentralised optimisation. In all these settings the presence of multiple learning agents renders the training problem non-stationary and often leads to unstable training or undesired final results. We present Learning with Opponent-Learning Awareness (LOLA), a method in which each agent shapes the anticipated learning of the other agents in the environment. The LOLA learning rule includes a term that accounts for the impact of one agent's policy on the anticipated parameter update of the other agents. Results show that the encounter of two LOLA agents leads to the emergence of tit-for-tat and therefore cooperation in the iterated prisoners' dilemma, while independent learning does not. In this domain, LOLA also receives higher payouts compared to a naive learner, and is robust against exploitation by higher order gradient-based methods. Applied to repeated matching pennies, LOLA agents converge to the Nash equilibrium. In a round robin tournament we show that LOLA agents successfully shape the learning of a range of multi-agent learning algorithms from literature, resulting in the highest average returns on the IPD. We also show that the LOLA update rule can be efficiently calculated using an extension of the policy gradient estimator, making the method suitable for model-free RL. The method thus scales to large parameter and input spaces and nonlinear function approximators. We apply LOLA to a grid world task with an embedded social dilemma using recurrent policies and opponent modelling. By explicitly considering the learning of the other agent, LOLA agents learn to cooperate out of self-interest. The code is at github.com/alshedivat/lola.

arXiv:1803.11485공식 코드 · Papers with Code

QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a network that estimates joint action-values as a complex non-linear combination of per-agent values that condition only on local observations. We structurally enforce that the joint-action value is monotonic in the per-agent values, which allows tractable maximisation of the joint action-value in off-policy learning, and guarantees consistency between the centralised and decentralised policies. We evaluate QMIX on a challenging set of StarCraft II micromanagement tasks, and show that QMIX significantly outperforms existing value-based multi-agent reinforcement learning methods.

arXiv:1802.05438공식 코드 · Papers with Code

Mean Field Multi-Agent Reinforcement Learning

Existing multi-agent reinforcement learning methods are limited typically to a small number of agents. When the agent number increases largely, the learning becomes intractable due to the curse of the dimensionality and the exponential growth of agent interactions. In this paper, we present \emph{Mean Field Reinforcement Learning} where the interactions within the population of agents are approximated by those between a single agent and the average effect from the overall population or neighboring agents; the interplay between the two entities is mutually reinforced: the learning of the individual agent's optimal policy depends on the dynamics of the population, while the dynamics of the population change according to the collective patterns of the individual policies. We develop practical mean field Q-learning and mean field Actor-Critic algorithms and analyze the convergence of the solution to Nash equilibrium. Experiments on Gaussian squeeze, Ising model, and battle games justify the learning effectiveness of our mean field approaches. In addition, we report the first result to solve the Ising model via model-free reinforcement learning methods.

arXiv:2502.00345v2cs.LG공식 코드 · Papers with Code

CTC: 협력적 다중 에이전트 강화학습을 위한 복합 과제 챌린지

CTC: The Composite Task Challenge for Cooperative Multi-Agent Reinforcement Learning

분업(division of labor, DOL)이 협력을 강화하는 데 핵심적인 역할을 한다는 점은 실제 응용에서 잘 알려져 있다. 이에 따라 많은 협력적 다중 에이전트 강화학습(MARL) 방법들이 에이전트 간 협력을 개선하기 위해 DOL 메커니즘을 도입해 왔다. 그러나 DOL과 협력을 평가하고 촉진하도록 특별히 설계된 벤치마크 과제가 부족하여, 협력적 MARL에서 이러한 메커니즘의 효과적인 개발과 배포가 제한되어 왔다. 현재의 협력적 MARL 방법과 실제 응용 사이의 이러한 격차는 DOL과 협력을 명시적으로 요구하는 평가 과제의 필요성을 부각시킨다. 이 격차를 해소하기 위해, 우리는 과제를 성공적으로 완료하기 위해 DOL과 협력을 모두 요구하도록 명시적으로 설계된 과제 모음인 복합 과제 챌린지(Composite Tasks Challenge, CTC)를 제안한다. CTC 과제는 두 가지 핵심 설계 원칙에 기반하여 구성된다: 1) DOL은 과제 성공의 필수 조건이다; 2) 어떤 원자적 하위 과제에서든 실패하면 전체 과제가 실패한다. 첫 번째 원칙은 DOL의 필요성을 강조하고, 두 번째 원칙은 협력의 중요성을 강제하여, CTC 과제에서 성공을 위해 두 구성 요소가 모두 필수적이 되도록 한다. 우리는 제안된 CTC 과제에 대해 9개의 대표적인 협력적 MARL 방법을 평가한다. 실험 결과, 모든 방법이 모든 CTC 과제에서 일관되게 0%의 테스트 승률을 기록하여, CTC 과제의 난이도와 현재 방법들의 한계를 부각시킨다. 향후 연구를 촉진하기 위해, 우리는 또한 모든 과제에서 0이 아닌 테스트 승률을 달성하는 가이드 솔루션을 제시하여 CTC 과제의 해결 가능성을 보여준다. 그러나 이 가이드 솔루션의 성능은 여전히 최적에 미치지 못하며, 이는 협력적 MARL 연구를 발전시키기 위한 도전적이고 의미 있는 테스트베드로서 CTC 과제의 가치를 더욱 부각시킨다.