Integrate the Reward, Don't Sample It: Window Likelihood for Interruptions and Backchannels in Full-Duplex Speech Models

원문 초록

When a user talks over a full-duplex dialogue model, it must decide whether the user is interrupting (stop) or backchannelling (carry on). Post-training can improve this discrimination or change how readily it stops. In our GRPO runs the second dominates: no configuration lowers both error rates on more than one model, and two released checkpoints discriminate no better than their bases. We trace this partly to how the reward is estimated. The probability of stopping inside a short window is a known function of a few stop probabilities the policy emits, yet GRPO estimates it by sampling. A group of rollouts that all agree gives no gradient, even when all fail; the push on the stop threshold is strongest where the model is unsure, whichever outcome is right; and whole-rollout credit reaches decisions the reward never judges. Confining the credit to the window removes only that leakage. Window likelihood (WL) samples the words but integrates over the stop decisions, so the probability of the correct outcome is computed exactly and its logarithm maximised. On HumDial-FDBench, across four open models, WL gives the best F1 and lowers both error rates by more than their confidence intervals, without talking into silence.