Stability Based Generalization Bounds for Exponential Family Langevin Dynamics
원문 초록
Recent years have seen advances in generalization bounds for noisy stochastic algorithms, especially stochastic gradient Langevin dynamics (SGLD) based on stability (Mou et al., 2018; Li et al., 2020) and information theoretic approaches (Xu and Raginsky, 2017; Negrea et al., 2019; Steinke and Zakynthinou, 2020). In this paper, we unify and substantially generalize stability based generalization bounds and make three technical contributions. First, we bound the generalization error in terms of expected (not uniform) stability which arguably leads to quantitatively sharper bounds. Second, as our main contribution, we introduce Exponential Family Langevin Dynamics (EFLD), a substantial generalization of SGLD, which includes noisy versions of Sign-SGD and quantized SGD as special cases. We establish data-dependent expected stability based generalization bounds for any EFLD algorithm with a O(1/n) sample dependence and dependence on gradient discrepancy rather than the norm of gradients, yielding significantly sharper bounds. Third, we establish optimization guarantees for special cases of EFLD. Further, empirical results on benchmarks illustrate that our bounds are non-vacuous, quantitatively sharper than existing bounds, and behave correctly under noisy labels.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Learning Non-Vacuous Generalization Bounds from Optimization
Learning Where to Learn: Training Distribution Selection for Provable OOD Performance
Operator Learning for Schrödinger Equation: Unitarity, Error Bounds, and Time Generalization
Graph Representational Learning: When Does More Expressivity Hurt Generalization?
Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws
Achieving Distributive Justice in Federated Learning via Uncertainty Quantification
토론
이 논문과 연결된 포럼 글이 아직 없습니다.