A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models
원문 초록
Beyond maximum likelihood estimation (MLE), the standard objective of a language model (LM) that optimizes good examples probabilities, many studies have explored ways that also penalize bad examples for enhancing the quality of output distribution, including unlikelihood training, exponential maximizing average treatment effect (ExMATE), and direct preference optimization (DPO). To systematically compare these methods and further provide a unified recipe for LM optimization, in this paper, we present a unique angle of gradient analysis of loss functions that simultaneously reward good examples and penalize bad ones in LMs. Through both mathematical results and experiments on CausalDialogue and Anthropic HH-RLHF datasets, we identify distinct functional characteristics among these methods. We find that ExMATE serves as a superior surrogate for MLE, and that combining DPO with ExMATE instead of MLE further enhances both the statistical (5-7%) and generative (+18% win rate) performance.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks
Making Language Model a Hierarchical Classifier and Generator
InstructFLIP: Exploring Unified Vision-Language Model for Face Anti-spoofing
Describe Anything Model for Visual Question Answering on Text-rich Images
Assay2Mol: large language model-based drug design using BioAssay context
Kodezi Chronos: A Debugging-First Language Model for Repository-Scale, Memory-Driven Code Understanding
토론
이 논문과 연결된 포럼 글이 아직 없습니다.