On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks
원문 초록
Kolmogorov--Arnold Networks (KANs), a recently proposed neural network architecture, have gained significant attention in the deep learning community, due to their potential as a viable alternative to multi-layer perceptrons (MLPs) and their broad applicability to various scientific tasks. Empirical investigations demonstrate that KANs optimized via stochastic gradient descent (SGD) are capable of achieving near-zero training loss in various machine learning (e.g., regression, classification, and time series forecasting, etc.) and scientific tasks (e.g., solving partial differential equations). In this paper, we provide a theoretical explanation for the empirical success by conducting a rigorous convergence analysis of gradient descent (GD) and SGD for two-layer KANs in solving both regression and physics-informed tasks. For regression problems, we establish using the neural tangent kernel perspective that GD achieves global linear convergence of the objective function when the hidden dimension of KANs is sufficiently large. We further extend these results to SGD, demonstrating a similar global convergence in expectation. Additionally, we analyze the global convergence of GD and SGD for physics-informed KANs, which unveils additional challenges due to the more complex loss structure. This is the first work establishing the global convergence guarantees for GD and SGD applied to optimize KANs and physics-informed KANs.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Prediction via Shapley Value Regression
Effective Probabilistic Time Series Forecasting with Fourier Adaptive Noise-Separated Diffusion
Symbolic Regression with Multimodal Large Language Models and Kolmogorov Arnold Networks
An Interval-Valued Time Series Forecasting Scheme With Probability Distribution Features for Electric Power Generation Prediction
MoFE-Time: Mixture of Frequency Domain Experts for Time-Series Forecasting Models
Decomposing the Time Series Forecasting Pipeline: A Modular Approach for Time Series Representation, Information Extraction, and Projection
토론
이 논문과 연결된 포럼 글이 아직 없습니다.