Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
원문 초록
Offline reinforcement learning has become one of the most practical RL settings. However, most existing works on offline RL focus on the standard setting with scalar reward feedback. It remains unknown how to universally transfer the existing rich understanding of offline RL from the reward-based to the preference-based setting. In this work, we propose a general framework to bridge this gap. Our key insight is transforming preference feedback to scalar rewards via binary reward labeling (BRL), and then any reward-based offline RL algorithms can be applied to the dataset with the reward labels. The information loss during the feedback signal transition is minimized with binary reward labeling in the practical learning scenarios. We theoretically show the connection between several recent PBRL techniques and our framework combined with specific offline RL algorithms. By combining reward labeling with different algorithms, our framework can lead to new and potentially more efficient offline PBRL algorithms. We empirically test our framework on preference datasets based on the standard D4RL benchmark. When combined with a variety of efficient reward-based offline RL algorithms, the learning result achieved under our framework is comparable to training the same algorithm on the dataset with actual rewards in many cases and better than the recent PBRL baselines in most cases.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimization
Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood
Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
Flow Q-Learning
Are Expressive Models Truly Necessary for Offline RL?
NetworkGym: Reinforcement Learning Environments for Multi-Access Traffic Management in Network Simulation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.