Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error
원문 초록
In this work, we study the use of the Bellman equation as a surrogate objective for value prediction accuracy. While the Bellman equation is uniquely solved by the true value function over all state-action pairs, we find that the Bellman error (the difference between both sides of the equation) is a poor proxy for the accuracy of the value function. In particular, we show that (1) due to cancellations from both sides of the Bellman equation, the magnitude of the Bellman error is only weakly related to the distance to the true value function, even when considering all state-action pairs, and (2) in the finite data regime, the Bellman equation can be satisfied exactly by infinitely many suboptimal solutions. This means that the Bellman error can be minimized without improving the accuracy of the value function. We demonstrate these phenomena through a series of propositions, illustrative toy examples, and empirical analysis in standard benchmark domains.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Shapley-Guided Utility Learning for Effective Graph Inference Data Valuation
Personalized Federated Collaborative Filtering: A Variational AutoEncoder Approach
Spatial-Division Augmented Occupancy Field for Bone Shape Reconstruction from Biplanar X-Rays
WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models
CompilerDream: Learning a Compiler World Model for General Code Optimization
ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast
토론
이 논문과 연결된 포럼 글이 아직 없습니다.