arXiv:2303.13465

Deep RL with Hierarchical Action Exploration for Dialogue Generation

Itsugun Cho, Ryota Takahashi, Yusaku Yanase, Hiroaki Saito

제출 2023-03-22

원문 초록

Traditionally, approximate dynamic programming is employed in dialogue generation with greedy policy improvement through action sampling, as the natural language action space is vast. However, this practice is inefficient for reinforcement learning (RL) due to the sparsity of eligible responses with high action values, which leads to weak improvement sustained by random sampling. This paper presents theoretical analysis and experiments that reveal the performance of the dialogue policy is positively correlated with the sampling size. To overcome this limitation, we introduce a novel dual-granularity Q-function that explores the most promising response category to intervene in the sampling process. Our approach extracts actions based on a grained hierarchy, thereby achieving the optimum with fewer policy iterations. Additionally, we use offline RL and learn from multiple reward functions designed to capture emotional nuances in human interactions. Empirical studies demonstrate that our algorithm outperforms baselines across automatic metrics and human evaluations. Further testing reveals that our algorithm exhibits both explainability and controllability and generates responses with higher expected rewards.

함께 살펴볼 논문

공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.

arXiv:2505.22442v4cs.LG공통 태그 Reinforcement Learning (RL), Offline RL공식 코드 · Papers with Code

Fully Offline Reinforcement Learning

토론

이 논문과 연결된 포럼 글이 아직 없습니다.