On the Sample Complexity of One Hidden Layer Networks with Equivariance, Locality and Weight Sharing
원문 초록
Weight sharing, equivariance, and local filters, as in convolutional neural networks, are believed to contribute to the sample efficiency of neural networks. However, it is not clear how each one of these design choices contributes to the generalization error. Through the lens of statistical learning theory, we aim to provide insight into this question by characterizing the relative impact of each choice on the sample complexity. We obtain lower and upper sample complexity bounds for a class of single hidden layer networks. For a large class of activation functions, the bounds depend merely on the norm of filters and are dimension-independent. We also provide bounds for max-pooling and an extension to multi-layer networks, both with mild dimension dependence. We provide a few takeaways from the theoretical results. It can be shown that depending on the weight-sharing mechanism, the non-equivariant weight-sharing can yield a similar generalization bound as the equivariant one. We show that locality has generalization benefits, however the uncertainty principle implies a trade-off between locality and expressivity. We conduct extensive experiments and highlight some consistent trends for these models.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systems
auto-fpt: Automating Free Probability Theory Calculations for Machine Learning Theory
Double descent in quantum machine learning
Gradient Equilibrium in Online Learning: Theory and Applications
NLPrompt: Noise-Label Prompt Learning for Vision-Language Models
Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
토론
이 논문과 연결된 포럼 글이 아직 없습니다.