PyTorchKR Papers

최신 논문

총 4편
태그·기간 등 상세 필터

카테고리 탐색

로그인 후 이 태그 구독

arXiv:2607.05866v1cs.LG

Differentially Private Natural Gradient Descent

차분 프라이버시(DP) 학습에서 DP-SGD의 기하학적 맹점(곡률 무시로 인한 비효율적 수렴)을 해결하기 위해, 공개 보조 데이터로 곡률을 추정하고 F⁻¹/²-whitened 공간에서 등방성 DP 연산을 비등방성 자연 경사 업데이트로 변환하는 DP-NGD 프레임워크를 제안한다. 동적 곡률 클램핑으로 학습 안정성을 확보한 DP-NGD는 표준 벤치마크에서 동일 프라이버시 예산 하에 최대 10배의 수렴 가속과 최신 정확도를 달성했다.

arXiv:2607.02194v1cs.LG

An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks

Physics-informed neural networks (PINNs) have emerged as a promising route to solve partial differential equations, yet they have struggled to reach the precision of classical solvers. The obstacle is increasingly understood to be one of optimisation, owing to the severely ill-conditioned loss landscape. We present $\textbf{DSGNAR}$: Doubly-Sketched Gauss-Newton with Adaptive Ratio, a scalable second-order optimisation framework that confronts this ill-conditioning and, in doing so, obtains unprecedented accuracy and speed. $\textbf{DSGNAR}$ couples a doubly-sketched Gauss-Newton model with a novel strategy that carefully controls both regularisation and step length. Across a suite of problems spanning nonlinear, chaotic, multi-scale, high-dimensional, and Navier-Stokes, the framework greatly improves on the state of the art: able to attain relative $\ell_2$ errors as low as $3\times10^{-16}$ in double precision, improve contemporary results by five orders of magnitude on the canonical Burgers' equation, and as much as eight orders on a high-dimensional Poisson problem, while remaining markedly faster. We further show that, in single precision, solutions at the limit of round-off error can be obtained very quickly: Burgers' equation to $\ell_2^{\text{rel}} = 4.75 \times 10^{-7}$ in under ten seconds. The framework is also robust to the choice of architecture, arithmetic precision, and initial hyperparameters. The code is available at https://www.github.com/wephy/physics-informed-neural-networks

arXiv:2508.12270v3cs.LG

L-SR1: Learned Symmetric-Rank-One Preconditioning

End-to-end deep learning has achieved impressive results but often relies on large labeled datasets, exhibits limited generalization to unseen scenarios, and incurs substantial computational cost. Classical optimization methods, in contrast, are more data-efficient and lightweight but frequently suffer from slow convergence. Learned optimizers aim to bridge this gap, yet existing approaches have focused primarily on first-order methods, while learned second-order optimization has received much less attention. We introduce L-SR1, a learned second-order optimizer inspired by the classical Symmetric Rank-One (SR1) method. At its core, L-SR1 employs a Projection-Guided Secant Mechanism (PGSM) that generates positive semi-definite preconditioners and biases meta-training toward the quasi-Newton secant relation. Through controlled analytic benchmarks, we study stability, generalization across problem dimensions, and search direction quality, and further evaluate L-SR1 on Monocular Human Mesh Recovery (HMR), where it outperforms both classical and learned optimization-based baselines. With a compact model and no reliance on task-specific fine-tuning or annotated data, L-SR1 demonstrates strong generalization and can be integrated into a broad range of iterative optimization problems to accelerate convergence and reduce the required number of iterations.

arXiv:2602.02016v2cs.LG

DASH: 배치 블록 사전조건화와 효율적인 역근 솔버를 통한 빠른 Shampoo

DASH: Faster Shampoo via Batched Block Preconditioning and Efficient Inverse-Root Solvers

Shampoo의 높은 계산 비용을 줄이기 위해 DASH(Distributed Accelerated Shampoo)가 제안되며, 사전조건자(preconditioner) 블록을 3D 텐서로 묶어 GPU 활용도를 높이고 Newton-DB 반복과 체비쇼프(Chebyshev) 다항식 근사로 역행렬 근 계산을 가속한다. GPU에 맞춘 구현은 Distributed Shampoo 대비 최대 5.6배 빠른 옵티마이저 스텝을 달성했고, Newton-DB는 테스트한 방법 중 반복당 검증 퍼플렉서티(perplexity)가 가장 낮았다.