On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluation
원문 초록
We study off-policy evaluation (OPE) in partially observable environments with complex observations, with the goal of developing estimators whose guarantee avoids exponential dependence on the horizon. While such estimators exist for MDPs and POMDPs can be converted to history-based MDPs, their estimation errors depend on the state-density ratio for MDPs which becomes history ratios after conversion, an exponential object. Recently, Uehara et al. [2022a] proposed future-dependent value functions as a promising framework to address this issue, where the guarantee for memoryless policies depends on the density ratio over the latent state space. However, it also depends on the boundedness of the future-dependent value function and other related quantities, which we show could be exponential-in-length and thus erasing the advantage of the method. In this paper, we discover novel coverage assumptions tailored to the structure of POMDPs, such as outcome coverage and belief coverage, which enable polynomial bounds on the aforementioned quantities. As a side product, our analyses also lead to the discovery of new algorithms with complementary properties.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effects
Trajectory World Models for Heterogeneous Environments
Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning
Minimum Empirical Divergence for Sub-Gaussian Linear Bandits
Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation
Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferences
토론
이 논문과 연결된 포럼 글이 아직 없습니다.