Correctness vs Efficiency: Behavior Control with Weight-Space Extrapolation in Code RL

원문 초록

A model trained with an imperfect reward in RL may learn to exploit the reward with undesired behavior. We study a controlled form of this problem in competitive programming. A program may pass small tests while violating the intended efficiency requirement on larger inputs. We construct a family of rewards that progressively relax the efficiency constraint, making the reward increasingly easy to exploit, and train separate policies. This sweep reveals a correctness-efficiency frontier. Relaxing the reward leads to less efficient programs and more resource-limit failures. Stricter rewards suppress these failures but increase correctness failures near the model's capability boundary. This frontier is also reflected in weight space. Interpolating between the resulting models reproduces the same behavioral change. More surprisingly, extrapolation in both directions further amplifies or suppresses it beyond the trained checkpoints. Extensive experiments show that this pattern holds across pure reasoning, tool use, and agentic coding, and at both 32B and 7B scales. This suggests a simple intervention for this form of reward hacking: use a loose reward to amplify the undesired behavior, use the weight-space difference to a strict-reward model as its steering vector, and then extrapolate past the strict model to suppress it.