Confidence Interval for Off-Policy Evaluation from Dependent Samples via Bandit Algorithm: Approach from Standardized Martingales
원문 초록
This study addresses the problem of off-policy evaluation (OPE) from dependent samples obtained via the bandit algorithm. The goal of OPE is to evaluate a new policy using historical data obtained from behavior policies generated by the bandit algorithm. Because the bandit algorithm updates the policy based on past observations, the samples are not independent and identically distributed (i.i.d.). However, several existing methods for OPE do not take this issue into account and are based on the assumption that samples are i.i.d. In this study, we address this problem by constructing an estimator from a standardized martingale difference sequence. To standardize the sequence, we consider using evaluation data or sample splitting with a two-step estimation. This technique produces an estimator with asymptotic normality without restricting a class of behavior policies. In an experiment, the proposed estimator performs better than existing methods, which assume that the behavior policy converges to a time-invariant policy.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effects
Trajectory World Models for Heterogeneous Environments
Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning
Minimum Empirical Divergence for Sub-Gaussian Linear Bandits
Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation
Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferences
토론
이 논문과 연결된 포럼 글이 아직 없습니다.