Exponential convergence rates for momentum stochastic gradient descent in the overparametrized setting
원문 초록
We prove explicit bounds on the exponential rate of convergence for the momentum stochastic gradient descent scheme (MSGD) for arbitrary, fixed hyperparameters (learning rate, friction parameter) and its continuous-in-time counterpart in the context of non-convex optimization. In the small step-size regime and in the case of flat minima or large noise intensities, these bounds prove faster convergence of MSGD compared to plain stochastic gradient descent (SGD). The results are shown for objective functions satisfying a local Polyak-Lojasiewicz inequality and under assumptions on the variance of MSGD that are satisfied in overparametrized settings. Moreover, we analyze the optimal choice of the friction parameter and show that the MSGD process almost surely converges to a local minimum.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Frictional Agent Alignment Framework: Slow Down and Don't Break Things
Physics-Informed Deep Learning for Nonlinear Friction Model of Bow-string Interaction
PyScrew: A Comprehensive Dataset Collection from Industrial Screw Driving Experiments
Barrier-Free Microhabitats: Self-Organized Seclusion in Microbial Communities
PUGS: Zero-shot Physical Understanding with Gaussian Splatting
Know Your Mistakes: Towards Preventing Overreliance on Task-Oriented Conversational AI Through Accountability Modeling
토론
이 논문과 연결된 포럼 글이 아직 없습니다.