On Compatible Generalization Bounds of Deep Residual Networks From a Dynamical System Perspective

원문 초록

Deep neural networks (DNNs) have significantly advanced machine learning, with model depth playing a central role in their successes. Deep residual networks (ResNets) fall into one of the most important classes of DNNs. In this paper, we study the compatible generalization behavior of deep residual networks with a broad class of activation functions in the deep-layer limit, from a dynamical system perspective. We first show a refined contraction inequality for Rademacher complexity of function classes composed by that type of activation functions. We then conduct an equivalence analysis of component hypothesis sets in Rademacher complexity and forward convergence analysis of network state variables, which enable us to establish compatible uniform generalization bounds that simultaneously achieve the optimal \(O(1/\sqrt{S})\) sample rate for both discrete- and continuous-time ResNets. The bounds also share a common layer-number independent leading term and non-positive refinement terms that capture the decomposition structure of the activation function, revealing a consistent generalization behavior in the deep-layer limit. Compared to previous works considering both discrete- and continuous-time ResNets simultaneously, our theoretical results not only have better sample rate, but also are established under weaker assumptions on the learnable parameters. We also support our theoretical findings with numerical experiments.