The Trajectory Alignment Coefficient in Two Acts: From Reward Tuning to Reward Learning

원문 초록

The success of RL is fundamentally tied to having a reward function that accurately reflects the task objective. Yet, designing reward functions is time-consuming and prone to misspecification. To address this, our first goal is to understand how to support RL practitioners in specifying reward functions. We leverage the Trajectory Alignment Coefficient (TAC), a metric that evaluates how closely a reward function's induced preferences match those of a domain expert. To test whether TAC provides meaningful support, we evaluated its impact in a user study where practitioners designed reward functions for Lunar Lander. We found that iterative TAC feedback allowed participants to design more performant reward functions while reporting lower cognitive workload than unguided reward design. However, the study also underscored that manual reward design, even with TAC, remains labor-intensive. This motivated our second goal: to learn a reward model that maximizes TAC directly. Specifically, we propose Soft-TAC, a differentiable approximation of TAC that can be used as a loss function to train reward models from human preferences. Our theoretical analysis guarantees that under Soft-TAC optimization: (1) the optimal reward function belongs exclusively to the set of human-aligned reward functions, and (2) this optimal reward remains invariant whether trained on clean or noisy preference data. Empirically, we first evaluate reward models trained with Soft-TAC across three robotic tasks, demonstrating that they consistently match or outperform those trained with the Cross-Entropy loss. Next, we evaluate Soft-TAC in the GT7 driving simulator, and find that Soft-TAC-trained reward models more successfully capture preference-specific driving styles. Ultimately, this work demonstrates that TAC can serve as both a practical tool for guiding reward design and a reward learning objective in complex domains.