원문 초록
The forward and reverse Kullback-Leibler (KL) divergences arise as limiting objectives in learning and inference, yet they induce markedly different inductive biases that cannot be explained by expectation terms alone. We introduce the Surprisal-R\'enyi Free Energy (SRFE), a normalized R\'enyi-Chernoff log-moment functional of the likelihood ratio. Although SRFE is algebraically related to R\'enyi divergence for fixed values of the order parameter, its endpoint-symmetric normalization yields a distinct free-energy viewpoint for analyzing how forward- and reverse KL behavior emerges near singular limits. We show that SRFE recovers the forward and reverse KL divergences as endpoint limits and derive local expansions around both regimes in which the variance of the log-likelihood ratio appears as a first-order correction away from the KL objective. These expansions reveal an explicit mean-variance tradeoff governing departures from KL-dominated behavior. We further establish a Gibbs-type variational characterization in terms of weighted KL projections and prove that SRFE controls large deviations of excess code length through Chernoff-type bounds, giving a Minimum Description Length interpretation of the functional. We evaluate SRFE on two synthetic settings (two-dimensional Gaussian-mixture variational inference and multi-class classification) and the long-tailed CIFAR-10 and CIFAR-100 (CIFAR-LT) datasets. On CIFAR-LT, retraining the best configuration of each loss over five seeds, SRFE matches cross-entropy accuracy on CIFAR-10-LT and gives mixed results on CIFAR-100-LT, improving few-shot accuracy at one of three imbalance factors. Together, these results position SRFE as a variance- and tail-sensitive, normalized R\'enyi-Chernoff free energy that clarifies the geometric, statistical, and large-deviation structure underlying the transition between forward- and reverse-KL limits.
제출 이력
확인된 버전만 표시합니다. 전체 제출 이력은 arXiv 원문에서 확인하세요.
- arXiv:2603.03405v2제출 시각 미확인
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
공통 태그로 연결된 공개 논문이 아직 없습니다.
토론
이 논문과 연결된 포럼 글이 아직 없습니다.