Bridging Offline and Online Reinforcement Learning for LLMs
원문 초록
We investigate the effectiveness of reinforcement learning methods for finetuning large language models when transitioning from offline to semi-online to fully online regimes for both verifiable and non-verifiable tasks. Our experiments cover training on verifiable math as well as non-verifiable instruction following with a set of benchmark evaluations for both. Across these settings, we extensively compare online and semi-online Direct Preference Optimization and Group Reward Policy Optimization objectives, and surprisingly find similar performance and convergence between these variants, which all strongly outperform offline methods. We provide a detailed analysis of the training dynamics and hyperparameter selection strategies to achieve optimal results. Finally, we show that multi-tasking with verifiable and non-verifiable rewards jointly yields improved performance across both task types.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling
TreeRL: LLM Reinforcement Learning with On-Policy Tree Search
VerIF: Verification Engineering for Reinforcement Learning in Instruction Following
Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
Learning Instruction-Following Policies through Open-Ended Instruction Relabeling with Large Language Models
No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
토론
이 논문과 연결된 포럼 글이 아직 없습니다.