Projected gradient methods for nonconvex and stochastic optimization: new complexities and auto-conditioned stepsizes
원문 초록
We present a novel class of projected gradient (PG) methods for minimizing a smooth but not necessarily convex function over a convex compact set. We first provide a novel analysis of the "vanilla" PG method, achieving the best-known iteration complexity for finding an approximate stationary point of the problem. We then develop an "auto-conditioned" projected gradient (AC-PG) variant that achieves the same iteration complexity without requiring the input of the Lipschitz constant of the gradient or any line search procedure. The key idea is to estimate the Lipschitz constant using first-order information gathered from the previous iterations, and to show that the error caused by underestimating the Lipschitz constant can be properly controlled. We then generalize the PG methods to the stochastic setting, by proposing a stochastic projected gradient (SPG) method and a variance-reduced stochastic gradient (VR-SPG) method, achieving new complexity bounds in different oracle settings. We also present auto-conditioned stepsize policies for both stochastic PG methods and establish comparable convergence guarantees.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Adaptive Semantic Token Communication for Transformer-based Edge Inference
Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Method
JaxSGMC: Modular stochastic gradient MCMC in JAX
SIM-Shapley: A Stable and Computationally Efficient Approach to Shapley Value Approximation
Adversarial Subspace Generation for Outlier Detection in High-Dimensional Data
CatCMA with Margin: Stochastic Optimization for Continuous, Integer, and Categorical Variables
토론
이 논문과 연결된 포럼 글이 아직 없습니다.