An Information-Theoretic Framework for Out-of-Distribution Generalization with Applications to Stochastic Gradient Langevin Dynamics
원문 초록
We study the Out-of-Distribution (OOD) generalization in machine learning and propose a general framework that establishes information-theoretic generalization bounds. Our framework interpolates freely between Integral Probability Metric (IPM) and $f$-divergence, which naturally recovers some known results (including Wasserstein- and KL-bounds), as well as yields new generalization bounds. Additionally, we show that our framework admits an optimal transport interpretation. When evaluated in two concrete examples, the proposed bounds either strictly improve upon existing bounds in some cases or match the best existing OOD generalization bounds. Moreover, by focusing on $f$-divergence and combining it with the Conditional Mutual Information (CMI) methods, we derive a family of CMI-based generalization bounds, which include the state-of-the-art ICIMI bound as a special instance. Finally, leveraging these findings, we analyze the generalization of the Stochastic Gradient Langevin Dynamics (SGLD) algorithm, showing that our derived generalization bounds outperform existing information-theoretic generalization bounds in certain scenarios.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Towards Robust Out-of-Distribution Generalization Bounds via Sharpness
Positional Encoding meets Persistent Homology on Graphs
Pruning Spurious Subgraphs for Graph Out-of-Distribtuion Generalization
Learning Where to Learn: Training Distribution Selection for Provable OOD Performance
The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants
LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders
토론
이 논문과 연결된 포럼 글이 아직 없습니다.