PyTorchKR Papers

최신 논문

총 2편
태그·기간 등 상세 필터

카테고리 탐색

로그인 후 이 태그 구독

arXiv:2606.11559v2cs.AI

Multi-Turn Agent Self-Distillation with Compressed Hindsight

In reinforcement learning for multi-turn agents, a sparse outcome reward makes credit assignment across intermediate tool calls difficult. On-policy self-distillation eases credit assignment by having the model teach itself with dense supervision: the model with privileged information (the teacher) scores each token generated by the model without it (the student). However, it remains unclear what forms of feedback are most effective in multi-turn agentic settings. We find that the natural multi-turn variant fails: when the teacher is given a complete successful demonstration at every turn, training makes the agent worse than the untrained model. The teacher ends up rewarding the student for copying the demonstration, even when a copied action is not right in the student's context. We propose HINT, in which a reflector, by default the policy itself, reads the completed trajectory and writes short turn-level feedback. The teacher then sees what went wrong at each turn, not the actions that followed. On Tau-Bench and WebShop, HINT matches GRPO with one-eighth as many rollouts.

arXiv:2606.23937v1cs.CL

When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents

τ-bench에서 Qwen2.5-3B/7B 분류기로 장기 도구 사용 에이전트의 사전 행동 정책 분류를 수행하며, exact-match retrieval recall이 검색된 정책의 하류 유용성을 대변하는지 검증했다. 정확한 governing clause의 rank-1 검색률은 7%에 불과하지만 검색 조항으로도 macro-F1 0.58을 달성해 gold 0.60과 유의미한 차이가 없었고, 구조화된 의사결정 상태는 튜닝 후 raw trajectory 대비 macro-F1을 0.13–0.17 높였다.