Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approach
원문 초록
In an Markov decision process (MDP), unobservable confounders may exist and have impacts on the data generating process, so that the classic off-policy evaluation (OPE) estimators may fail to identify the true value function of the target policy. In this paper, we study the statistical properties of OPE in confounded MDPs with observable instrumental variables. Specifically, we propose a two-stage estimator based on the instrumental variables and establish its statistical properties in the confounded MDPs with a linear structure. For non-asymptotic analysis, we prove a $\mathcal{O}(n^{-1/2})$-error bound where $n$ is the number of samples. For asymptotic analysis, we prove that the two-stage estimator is asymptotically normal with a typical rate of $n^{1/2}$. To the best of our knowledge, we are the first to show such statistical results of the two-stage estimator for confounded linear MDPs via instrumental variables.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effects
Trajectory World Models for Heterogeneous Environments
Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning
Minimum Empirical Divergence for Sub-Gaussian Linear Bandits
Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation
Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferences
토론
이 논문과 연결된 포럼 글이 아직 없습니다.