We study ordinary kernel ridge regression over full H\"older-Zygmund balls, measuring errors in Sobolev norms and in pointwise and uniform losses for mixed derivatives. For targets of smoothness $s$, every RKHS norm equivalent to $H^{s+d/2}$ gives the optimal $H^r$ risk for $0\le r<s$; integer choices of $r$ include integrated squared derivative losses. Pointwise and uniform guarantees require additional kernel structure. A localized wavelet kernel attains the sharp derivative rates with the same regularization orders as for function estimation, whereas equivalent changes of the RKHS norm can destroy the population bias rates. For the ordinary Gaussian kernel with a shrinking bandwidth, we prove a uniform derivative risk bound with the optimal polynomial rate and an explicit, unoptimized logarithmic factor. The results allow bounded measurable design densities bounded away from zero and conditionally sub-Gaussian noise.
We develop a quantitative statistical theory of transformers in the large-context regime by adopting the abstraction of contextual flow maps (CFMs): dynamical systems that evolve a distinguished token in the presence of a contextual measure across a stack of attention blocks. Within this framework, the finite-context model approximates an idealized infinite-context system in which the contextual measure is replaced by its underlying population, so that the context length $n$ becomes a statistical resource. Exploiting the McKean--Vlasov structure of the dynamics and the classical machinery of propagation of chaos, we establish a forward bound controlling the deviation between the finite- and infinite-context CFMs uniformly along depth, and a backward bound controlling the deviation between the corresponding training trajectories uniformly across iterations of online gradient descent. Both bounds achieve the optimal Wasserstein rate $n^{-1/d}$ for general CFMs and parametric rate $n^{-1/2}$ for a restricted class of CFMs that includes transformers as a special case. The analysis rests on a new Eulerian adjoint formulation of the loss gradient and stability estimates for the resulting forward--adjoint system, both of which may be of independent interest.
In recent years, Rectified flow (RF) has gained considerable popularity largely due to its generation efficiency and state-of-the-art performance. In this paper, we investigate how well RF automatically adapts to the intrinsic low dimensionality of the support of the target distribution to accelerate sampling. We show that, using a carefully designed \blue{end-point adapted} time-discretization scheme and with sufficiently accurate drift estimates, the RF sampler enjoys an iteration complexity of order $O(k/\varepsilon)$ (up to log factors), where $\varepsilon$ is the precision in total variation distance and $k$ is the intrinsic dimension of the target distribution. In addition, we show that the denoising diffusion probabilistic model (DDPM) procedure is equivalent to a stochastic version of RF by establishing a novel connection between these processes and stochastic localization. Building on this connection, we further design a stochastic RF sampler that also adapts to the low-dimensionality of the target distribution under mild requirements on the accuracy of the drift estimates, and also with a specific time schedule. We illustrate the efficacy of newly designed time-discretization schedules with simulations on the synthetic data and text-to-image (T2I) data experiments.
Inference-time alignment for diffusion models aims to adapt a pre-trained reference diffusion model toward a target distribution without retraining the reference score network, thereby preserving the generative capacity of the reference model while enforcing desired properties at inference time. A central mechanism for achieving such alignment is guidance, which modifies the sampling dynamics through an additional drift term. In this work, we introduce variationally stable Doob's matching, a novel framework for provable guidance estimation grounded in Doob's $h$-transform. Our approach formulates guidance as the gradient of the logarithm of an underlying Doob's $h$-function and employs gradient-regularized regression to simultaneously estimate both the $h$-function and its gradient, resulting in a consistent estimator of the guidance. Theoretically, we establish non-asymptotic convergence rates for the estimated guidance. Moreover, we analyze the resulting controllable diffusion processes and prove non-asymptotic convergence guarantees for the generated distributions in the Wasserstein-2 distance. Finally, we show that variationally stable guidance estimators are adaptive to unknown low dimensionality, effectively mitigating the curse of dimensionality under low-dimensional linear subspace assumptions. Numerical experiments demonstrate the effectiveness of our proposed method and show the advantages compared to the baselines.
We study the sequential root-finding problem with repeated noisy measurements. We propose sequential-test bisection STB, which uses a nonparametric sequential sign test to determine when to move to a new design point. Compared with variants of stochastic approximation, STB offers broader adaptivity and faster convergence. When the regression function satisfies $|f(x)|\geq \textsf{c}_0|x-\theta|^\gamma$ in a neighborhood of $\theta$ for constants $\textsf{c}_0,\gamma>0$ where $\theta$ is the root, i.e. $f(\theta) = 0$, we establish non-asymptotic guarantees showing that STB, without using the knowledge of $\gamma$, achieves the minimax adaptive optimal convergence rate $\widetilde{O}(n^{-1/(2\gamma)})$, outperforming Robbins--Monro and related stochastic-approximation methods in convergence rate. Under sub-Gaussian noise, STB provides finite-sample confidence sequences for the root $\theta$ without estimating a derivative. In the positive-slope model, we augment STB with {randomized weighted-section and a line fitted to all observations}, leading to asymptotic normality and consistent variance estimation. We formalize the efficiency by a convolution theorem of H\'ajek-Le Cam type. Furthermore, when the regression function is discontinuous at the root, we show that Robbins--Monro converges at rate $1/n$, whereas STB converges super-polynomially. We extend this construction to multivariate nonparametric level-set estimation by sampling along rays. As a technical byproduct, we establish a new central limit theorem under random stopping, with proof techniques that may be of independent interest.
Variational inference, as an alternative to Markov chain Monte Carlo sampling, has played a transformative role in enabling scalable computation for complex Bayesian models. Nevertheless, existing approaches often depend on either rigid model-specific formulations or stochastic black-box optimization routines. Tangent approximation is a principled class of structured variational methods that exploit the geometry of the underlying probability model. However, its utility has largely been confined to logistic regression and related modeling regimes. In this article, we propose a novel variational framework based on tangent transformation for a broad class of probability models characterized by strongly super-Gaussian likelihoods. Our method leverages convex duality to construct tangent minorants of the log-likelihood, thereby inducing conjugacy with Gaussian priors over model parameters in an otherwise intractable setup. Under mild assumptions on the data-generating mechanism, we establish algorithmic convergence guarantees, a contribution that stands in contrast to the limited theoretical assurances typically available for black-box variational methods. Additionally, we derive near-parametric variational risk bounds. Superior performance of our proposed methodology is illustrated on simulated and real-data scenarios that challenge state-of-the-art variational algorithms in terms of scalability and their ability to consistently capture complex underlying data structure.
Objective functions based on Hellinger distance yield robust and efficient estimators of model parameters. Motivated by privacy and regulatory requirements encountered in contemporary applications, we derive in this paper \emph{private minimum Hellinger distance estimators}. The estimators satisfy a new privacy constraint, namely, Hellinger differential privacy, while retaining the robustness and efficiency properties. We demonstrate that Hellinger differential privacy shares several features of standard differential privacy while allowing for sharper inference. Additionally, for computational purposes, we also develop Hellinger differentially private gradient descent and Newton-Raphson algorithms. We illustrate the behavior of our estimators in finite samples using numerical experiments and verify that they retain robustness properties under gross-error contamination.
We study the subspace-constrained Tyler's estimator (STE) for recovering a low-dimensional linear subspace from data contaminated by outliers. Under deterministic conditions on inlier spread and outlier alignment, we prove linear convergence to the true subspace from a sufficiently accurate initialization. The guarantee also applies when the dimension-scaled signal-to-noise ratio is below one, where general robust subspace recovery is SSE-hard. When initializing STE from a candidate subspace, we provide a sufficient recovery condition based on the initial angular error and the size of the identity shift applied to the subspace projector. For noisy inliers, we prove a subspace error proportional to the noise level within an explicit iteration bound. Finally, under an asymptotic, anisotropic Gaussian inlier--outlier model, we show that Tyler's M-estimator (TME) provides a successful STE initialization even when TME's own leading eigenspace differs from the target subspace.
We consider nonparametric maximum likelihood estimation of the intensity function of a Poisson process, allowing the intensity to depend on observed covariates. Such models describe how event rates vary with external conditions, including conditions under which the intensity becomes arbitrarily small or vanishes. However, risk analysis based on the Kullback-Leibler (KL) divergence typically requires the true intensity to be bounded away from zero, a restrictive assumption in applications. We develop a unified approach that avoids this assumption and derive an oracle inequality for the covariate-averaged Hellinger distance between the true intensity and the nonparametric maximum likelihood estimator (NPMLE). As an application of the oracle inequality, we establish convergence rates for NPMLEs based on deep neural networks (DNNs). Our results show that, as in nonparametric regression, the ability of DNNs to capture underlying structure leads to improved convergence rates. Specifically, DNN-based NPMLEs mitigate the curse of dimensionality when the true intensity has a compositional structure and automatically adapt to a low-dimensional Riemannian manifold structure of the event space. For intensity functions with a compositional structure, DNN-based NPMLEs achieve nearly minimax-optimal convergence rates without explicitly incorporating this structure into the network architecture. Simulations demonstrate that DNN-based estimators outperform kernel estimators in most settings, while manifold kernels remain competitive in some designs. Finally, we analyze stimulus-evoked neural spiking using data from the Allen Brain Observatory Visual Coding Neuropixels project. DNN-based NPMLEs provide better out-of-sample performance than a conditional kernel estimator, as evaluated by held-out log-likelihood.
Instance-dependent limits for bandit regret, sequential tests, best-arm identification and Markov decision processes are proved by one chain. It runs from a divergence decomposition through data processing to a statistic, a testing step that turns a class guarantee into a requirement on the history, and optimization over alternatives and allocations. Only the first step is an identity. This work writes the chain as an exact ledger. By the chain rule, data processing loses exactly a conditional divergence, which vanishes for arms in natural exponential families at the per-arm counts and reward sums. A one-parameter family of martingale identities extends the decomposition to every tilt between two instances. The classical limits follow by relaxations with named defects. All defects but the asymptotic step's can be computed from an algorithm's own runs, the uniform quantification's only when the class guarantee is a number. On seven published sources, from movie ratings to engine replacement, the ledger locates the difference between standard algorithms and the classical limits. For identification among trained models and for Wald's test the largest term is usually the uniform quantification's.
Learning a shared subspace in multitask learning is an important statistical problem, yet exact rates remain poorly understood in many of the formal settings considered in the literature. To address this gap, which arises from limitations of existing upper-bound techniques, we propose a general multitask lemma. The lemma provides sufficient conditions for fast subspace estimation in generic multitask settings, in the form of geometric conditions on the task regression functions and their estimators. We subsequently show that the multitask lemma yields sharp minimax rates---previously unknown---for linear regression and classification in the regimes considered, and rates matching up to logarithmic factors for additive-noise regression.
We extend the recently introduced e-closure principle for multiple testing to general state and action spaces and to arbitrary, including continuously indexed, families of risks. A collection of e-values indexed by states and risks certifies a menu of actions, and every data-dependent selection from this menu controls each expected loss at its prescribed level. Conversely, every valid procedure is certified by its own normalized losses, which often leave part of the risk budget unused. We introduce a method called recertification, which spends this unused budget. Using known distributional structure, it enlarges the certifying e-values while preserving validity and selects a less conservative action from the enlarged menu. One particularly compelling application is to the popular framework of conformal risk control (CRC) with a known loss family. Here, recertification redistributes unused risk budget among the held-out points of each hypothetical augmented sample. The resulting controller is valid under every finite exchangeable law and less (usually strictly) conservative than CRC. For convex losses on an interval of actions, repeated recertification converges geometrically to an admissible controller. As a second example involving false discovery rate control with e-values, recertifying randomized e-BH yields a closed-form procedure that rejects a superset of randomized e-BH for every realization of the randomization and maximizes the expected number of rejections under the same constraints. Recertifying with the larger e-collection of closed e-BH gives a second closed-form procedure that rejects a superset of the first. Experiments on CIFAR-100 hierarchical classification and simulations demonstrate improved budget utilization and gains in informativeness and power over the corresponding baselines.
Accurate and efficient representation of non-Gaussian state distributions is critical for estimation, control, and decision-making, especially when the system is highly nonlinear. This paper develops a mathematical framework for estimating non-Gaussian uncertainty via semi-nonparametric (SNP), or Gallant-Nychka, densities. SNP densities use squared probabilists' Hermite polynomials to guarantee nonnegative probability densities while representing non-Gaussian behavior. For uncertainty propagation, once fitted to propagated state samples, SNP densities enable direct analytical density and probability evaluations without needing to tune additional hyperparameters besides the polynomial order. To enable their application to nonlinear dynamical systems, we derive analytic marginal and cumulative distribution functions and develop a lossless convex formulation of the coefficient estimation problem along a given sign branch. The approach is demonstrated on the chaotic Lorenz system and compared against Gaussian mixture models and Edgeworth approximations. Results demonstrate that SNP densities can accurately represent non-Gaussian state distributions while enabling direct probability and quantile evaluation, making them a promising tool for density estimation and uncertainty quantification in nonlinear estimation and control.
We show that outlier eigenvectors of the Bethe Hessian achieve weak recovery in the stochastic block model down to the Kesten-Stigum threshold, for every fixed common expected degree $d>1$. This extends previous recovery guarantees, which required $d$ to be sufficiently large. This threshold is information-theoretically optimal for the symmetric two-community model and is conjectured to mark the limit of efficient recovery in broader symmetric models. Our result matches the recovery guarantees of non-backtracking spectral methods while using Hermitian $n\times n$ matrices. To do this, we identify recursive formulas for the negative outlier eigenvalues and show that their eigenspaces have nonvanishing overlap with the corresponding planted eigenspaces. The proof constructs approximate eigenvectors from the planted labels on the limiting Poisson tree, with weights given by its Green's function, and transfers these approximations to the finite graph.
We study high-dimensional graph combinatorial hypothesis testing problems in which a signal is supported on the edge set of an unknown subgraph drawn from a class invariant under vertex relabeling. For Gaussian observations, we characterize sharp statistical and computational thresholds (at polynomial order) for the signal magnitude required for strong detection across various combinatorial graph alternatives. Our analysis is based on bounding the second moments of the full and low-degree likelihood ratios in terms of the overlap of two independent latent structures and comparing the resulting lower bounds with scan, average, and coordinatewise maximum tests. Under an appropriate low-degree conjecture for computational hardness, we obtain planted-clique thresholds matching those known for Gaussian submatrix detection and identify new statistical-computational gaps for multiple planted cliques and hypercliques, thick cycles and thick paths. We also establish a novel transfer principle across natural exponential family observation models: matching the one-coordinate chi-square divergence preserves full likelihood-ratio second moments exactly and makes low-degree second moments equivalent up to constant factors under a degree condition. Under local regularity, we obtain corresponding testing upper bounds in terms of an effective signal strength. When these upper and lower bounds match, they yield universal threshold scales in the local alternative regime. As a consequence, statistical-computational gaps for planted-clique detection in Erd\H{o}s--R\'enyi settings can be deduced from the Gaussian model.
This paper studies robust uncertainty quantification for separable nonlinear least squares (SNLLS) problems. We propose two identification methods, (i) Residual-Perturbed Gradient (RPG) and (ii) Global-Fit-Gap (GFG), to construct joint distribution-free confidence regions for both the linear and nonlinear parameters. By exploiting assumed group invariance of the noise (such as symmetry or exchangeability), we apply random perturbations (such as sign-changes or permutations) to build the regions based on resampling and ranking. We prove that both methods guarantee non-asymptotically exact coverage (type I error). Finally, we illustrate the constructions through two numerical experiments: (i) identifying generalized finite impulse response (GFIR) systems using Laguerre polynomials with an adjustable decay rate and (ii) fitting Gaussian radial basis function (RBF) models with variable knots and widths.
Neural networks are powerful memorizers which are often able to interpolate their training data even when the labels have been randomized. While there exists an extensive literature on the memorization capacity of fully connected networks, in practice more specific architectures with problem-dependent properties are preferred, such as convolutional networks for image classification. In this paper we consider convolutional networks which are invariant with respect to a given arbitrary finite group of orthogonal transformations, which may include translations, reflections, and/or rotations of images and invariances of higher order tensors. We prove upper bounds on the size of an invariant convolutional neural network with threshold activations that is required to interpolate a given dataset with two classes. Our bounds are formulated in terms of a complexity measure that takes the invariant structure into account and is instance-specific, i.e., it is related to the mutual arrangement of the classes in the given dataset and their geometric properties rather than the number of samples. Our proof is constructive: for given input data, instances of an interpolating network can be computed via a randomized algorithm with high probability. In numerical experiments we investigate the practical effectiveness of this algorithm.
We consider the tensor principal components analysis (tensor PCA) problem on the sphere, in which we are asked to recover an unknown vector in $\mathcal{S}_N=\{x\in\mathbb{R}^N:\lVert x\rVert^2=N\}$ from a noisy observation of its $k$-th tensor power, for a fixed $k\ge3$. Tensor PCA exhibits a well-studied information-computation gap: global algorithms such as spectral methods and sum-of-squares work above an algorithmic threshold, below which we know matching lower bounds (against these and some other restricted models of computation). In contrast, local algorithms such as Markov chains were only known to succeed from a random initialization at a polynomially larger signal-to-noise ratio, and were believed to be inferior to global algorithms for tensor PCA. Continuing the program initiated in Part I [arXiv:2610.16705], we show that spherical Langevin dynamics, when supplemented with annealing and ergodic averaging, achieves polynomial-time strong recovery above the algorithmic threshold from a uniformly random initialization. Together with Part I, our results demonstrate that local Markov chains can succeed all the way down to the algorithmic threshold from an uninformative cold start.
We study conformal prediction under covariate shift with a known covariate density ratio, seeking prediction sets with minimum expected length under the target distribution. For every calibration sample size, we prove that the rule combining the inverse conditional-density score with randomized weighted conformal calibration minimizes this expected length among all rules valid over the full weighted-exchangeability class. We approximate the oracle using quantile and kernel histograms, including forest-weighted estimators, and select estimators and tuning parameters by resampling to estimate calibrated target length. On a bounded response domain, we bound excess length linearly in $L^1$ density error at fixed calibration size, without requiring a uniform positive lower bound on the true density. We also bound selection-pool and resampling errors. Under explicit growth conditions, the selected procedure approaches the same oracle while the calibration size remains fixed. Experiments on 36 known-density configurations and four public regression datasets compare coverage, prediction-set length, and runtime with existing methods. In targeted low-coverage and strong-shift comparisons, inverse-density selection yields shorter mean sets in most cases and lower mean runtime than the evaluated weighted baselines, with some differences in achieved coverage.
For each time $t$, the regression function $m_t(x)=E[Y_t\mid X_t=x]$ of an outcome on contemporaneous covariates can be regarded as a field on covariate space. When the covariates of units move as a diffusion and each unit's outcome has its own dynamics, we show that $m_t$ satisfies a second-order evolution equation whose diffusion term is $\frac12\mathrm{tr}(\Sigma\nabla^2 m_t)$, the Hessian of the regression function weighted by the covariance $\Sigma$ of covariate increments, whose transport term is the drift of the time-reversed covariate process, and whose source term is the conditional mean rate of change of the outcome; this yields a Feynman-Kac representation along reversed covariate paths. We relate this Hessian to quadratic regression (under Gaussian design the best quadratic approximation has Hessian $E[\nabla^2 m(X)]$) and to kernel smoothing bias. For bistable replicator dynamics on covariate space, travelling fronts are exactly logistic in $(x,t)$, so one logistic regression identifies the diffusion-reaction ratio and the invading strategy. We then ask whether spread in covariate space reflects fluctuating unit characteristics or imitation among similar units. Imitation acts through exactly the same operator as covariate fluctuation, so the two are observationally equivalent in the regression field; panel data separate them because they identify $\Sigma$ directly. The two cannot be separated on a travelling front, and we propose a split-sample instrumental-variables test that works on seeded, non-front profiles, with size and power evaluated in agent-based experiments.