원문 초록
Preference learning is a powerful tool for reward learning, but treats the teacher as a passive oracle that answers queries driven by the learner. This learner-guided query selection ignores the teacher's knowledge of the solution. We reverse the typical preference query selection method by having the teacher create ordered preference pairs, since a teacher who knows the target can choose more informative pairs than a learner choosing its own queries. Exploiting this advantage requires the teacher to model what the learner knows, and that model goes stale when the learner's belief changes in ways the teacher cannot see, as when several teachers each teach without knowing what the others taught. We call this teacher-model drift. In this work, we use second-order Theory-of-Mind (ToM-2) to have the learner create understanding statements that correct the teacher's model of the learner's knowledge. In simulation, we show that a learner with a perfect teacher, one with perfect knowledge of the learner, needs about half as many rounds as learner-guided querying to learn the reward. Yet when teacher-model drift is induced by alternating between four co-teachers, the advantage erodes. We tested two types of understanding statement generation: the learner's mean belief and ToM-2 drift minimization. ToM-2 statements reduced the drift and recovered more of the lost advantage than mean statements. In a within-subjects study (n = 101) with a simulated ideal co-teacher, ToM-2 statements raised the learner's final alignment with the true reward above both uninformed and mean-informed teaching; on the regret participants removed in their own rounds, they outperformed uninformed teaching but not mean statements.
제출 이력
확인된 버전만 표시합니다. 전체 제출 이력은 arXiv 원문에서 확인하세요.
- arXiv:2608.11229v2제출 시각 미확인
- arXiv:2608.11229v1제출 시각 미확인
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
공통 태그로 연결된 공개 논문이 아직 없습니다.
토론
이 논문과 연결된 포럼 글이 아직 없습니다.