Coordinatewise Epsilon for Adam under Heavy-Tailed Noise

원문 초록

Adam estimates second moments coordinatewise, but typically adds the same scalar $\varepsilon$ to every denominator. We study how this shared $\varepsilon$ affects convergence when noise and curvature vary across coordinates. Under coordinatewise finite-$p$ noise moments, $p\in(1,2]$, choosing $\varepsilon_i$ from the noise and curvature bounds for each coordinate attains the heavy-tailed $\ell_1$-stationarity benchmark $O(T^{-\frac{p-1}{3p-2}})$, matching the known optimal time exponent for smooth nonconvex stochastic optimization. For $1<p<2$, Adam with a shared $\varepsilon_t$ admits a lower bound of order $\Omega(T^{-\frac{p-1}{2p}})$. This lower bound holds for every predictable scalar choice of $\eta_t$, $\beta_{1,t}$, $\beta_{2,t}$, and $\varepsilon_t$, precluding a dimension-uniform benchmark rate for this class.