Norm-based Generalization Bounds for Compositionally Sparse Neural Networks
원문 초록
In this paper, we investigate the Rademacher complexity of deep sparse neural networks, where each neuron receives a small number of inputs. We prove generalization bounds for multilayered sparse ReLU neural networks, including convolutional neural networks. These bounds differ from previous ones, as they consider the norms of the convolutional filters instead of the norms of the associated Toeplitz matrices, independently of weight sharing between neurons. As we show theoretically, these bounds may be orders of magnitude better than standard norm-based generalization bounds and empirically, they are almost non-vacuous in estimating generalization in various simple classification problems. Taken together, these results suggest that compositional sparsity of the underlying target function is critical to the success of deep neural networks.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Learning Non-Vacuous Generalization Bounds from Optimization
Learning Where to Learn: Training Distribution Selection for Provable OOD Performance
Operator Learning for Schrödinger Equation: Unitarity, Error Bounds, and Time Generalization
Graph Representational Learning: When Does More Expressivity Hurt Generalization?
Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws
Achieving Distributive Justice in Federated Learning via Uncertainty Quantification
토론
이 논문과 연결된 포럼 글이 아직 없습니다.