Variance-Reducing Couplings for Random Features
원문 초록
Random features (RFs) are a popular technique to scale up kernel methods in machine learning, replacing exact kernel evaluations with stochastic Monte Carlo estimates. They underpin models as diverse as efficient transformers (by approximating attention) to sparse spectrum Gaussian processes (by approximating the covariance function). Efficiency can be further improved by speeding up the convergence of these estimates: a variance reduction problem. We tackle this through the unifying lens of optimal transport, finding couplings to improve RFs defined on both Euclidean and discrete input spaces. They enjoy theoretical guarantees and sometimes provide strong downstream gains, including for scalable approximate inference on graphs. We reach surprising conclusions about the benefits and limitations of variance reduction as a paradigm, showing that other properties of the coupling should be optimised for attention estimation in efficient transformers.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Statistical Machine Learning for Astronomy -- A Textbook
LLINBO: Trustworthy LLM-in-the-Loop Bayesian Optimization
MathOptAI.jl: Embed trained machine learning predictors into JuMP models
Tailored Architectures for Time Series Forecasting: Evaluating Deep Learning Models on Gaussian Process-Generated Data
The Currents of Conflict: Decomposing Conflict Trends with Gaussian Processes
Few-Shot Speech Deepfake Detection Adaptation with Gaussian Processes
토론
이 논문과 연결된 포럼 글이 아직 없습니다.