Gap-Dependent Bounds for Federated $Q$-learning
원문 초록
We present the first gap-dependent analysis of regret and communication cost for on-policy federated $Q$-Learning in tabular episodic finite-horizon Markov decision processes (MDPs). Existing FRL methods focus on worst-case scenarios, leading to $\sqrt{T}$-type regret bounds and communication cost bounds with a $\log T$ term scaling with the number of agents $M$, states $S$, and actions $A$, where $T$ is the average total number of steps per agent. In contrast, our novel framework leverages the benign structures of MDPs, such as a strictly positive suboptimality gap, to achieve a $\log T$-type regret bound and a refined communication cost bound that disentangles exploration and exploitation. Our gap-dependent regret bound reveals a distinct multi-agent speedup pattern, and our gap-dependent communication cost bound removes the dependence on $MSA$ from the $\log T$ term. Notably, our gap-dependent communication cost bound also yields a better global switching cost when $M=1$, removing $SA$ from the $\log T$ term.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Q-Learning under Finite Model Uncertainty
Personalized Exercise Recommendation with Semantically-Grounded Knowledge Tracing
ADDQ: Adaptive Distributional Double Q-Learning
Inverse Q-Learning Done Right: Offline Imitation Learning in $Q^π$-Realizable MDPs
Distributionally Robust Deep Q-Learning
A critical assessment of reinforcement learning methods for microswimmer navigation in complex flows
토론
이 논문과 연결된 포럼 글이 아직 없습니다.