Iterative Policy Optimization: Batch-Online Multi-Turn RL via Single-Turn RL

원문 초록

Practical LLM agents often operate over multi-turn conversations where success is determined only after the full interaction ends. Most multi-turn RL methods train via on-policy rollouts, but unlike in single-turn RL, the policy cannot produce a trajectory alone, since an external environment must respond after each agent turn. For conversational agents, this environment is a user, but real users are generally unavailable inside the training loop and simulated users are difficult to build faithfully. Separately, real-world deployment is rarely fully online or fully offline. The common production pattern is called batch online, where the current policy is deployed to collect a batch of interaction data, then retrained on that batch and redeployed. We show that this batch-online setting creates an opportunity for applying the classical approximate policy iteration algorithm. Our central observation is that running standard token-level, single-turn GRPO with a learned turn-level $Q$-function as the reward model is a policy improvement step for the multi-turn problem. Building on this, we present Iterative Policy Optimization (IterPO), which alternates between fitting $Q^\pi$ from logged Monte Carlo returns via standard reward modeling (policy evaluation) and running single-turn GRPO against $Q^\pi$ (policy improvement). During policy improvement, the learned $Q^\pi$ scores candidate responses by their expected downstream return, avoiding the need for simulator rollouts of the remaining conversation. Overall, this allows us to do principled multi-turn RL using only single-turn RL methods without an interactive environment or user simulator inside the training loop. We demonstrate the effectiveness of IterPO on six multi-turn negotiation environments.