Revisiting the acceleration phenomenon via high-resolution differential equations
원문 초록
Nesterov's accelerated gradient descent (NAG) is one of the milestones in the history of first-order algorithms. It was not successfully uncovered until the high-resolution differential equation framework was proposed in [Shi et al., 2022] that the mechanism behind the acceleration phenomenon is due to the gradient correction term. To deepen our understanding of the high-resolution differential equation framework on the convergence rate, we continue to investigate NAG for the $\mu$-strongly convex function based on the techniques of Lyapunov analysis and phase-space representation in this paper. First, we revisit the proof from the gradient-correction scheme. Similar to [Chen et al., 2022], the straightforward calculation simplifies the proof extremely and enlarges the step size to $s=1/L$ with minor modification. Meanwhile, the way of constructing Lyapunov functions is principled. Furthermore, we also investigate NAG from the implicit-velocity scheme. Due to the difference in the velocity iterates, we find that the Lyapunov function is constructed from the implicit-velocity scheme without the additional term and the calculation of iterative difference becomes simpler. Together with the optimal step size obtained, the high-resolution differential equation framework from the implicit-velocity scheme of NAG is perfect and outperforms the gradient-correction scheme.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Scaling Laws of Dynamic High-Capacity Ride-Sharing
CUTS+: High-dimensional Causal Discovery from Irregular Time-series
CosmoPower-JAX: high-dimensional Bayesian inference with differentiable cosmological emulators
HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion
MIReAD: Simple Method for Learning High-quality Representations from Scientific Documents
High-Fidelity 3D Face Generation from Natural Language Descriptions
토론
이 논문과 연결된 포럼 글이 아직 없습니다.