We prove $F_{\mathrm{weighted}}(2,2,3)<11/6$ for minimum-Frobenius-norm least-squares regression with two input and two output coordinates, improving the upper bound $15/8$. More precisely, a universal $\varepsilon_0>0$ satisfies $\mathcal T_3\le(5/6-\varepsilon_0)E$ for every finite residual frame. The proof combines an exact scalar certificate, rigidity of equality in planar selection, and a compactness argument that controls disappearing weights. Together with the known lower bound, this gives $13/8\le F_{\mathrm{weighted}}(2,2,3)<11/6$. No numerical value of $\varepsilon_0$ is claimed.
Uncertainty quantification in prediction is a central problem in statistics and machine learning. Conformal prediction provides a flexible approach to constructing prediction sets with valid marginal coverage under exchangeability. In this paper, we investigate conformal prediction for both one-step and multi-step forecasting from a single stationary time series, addressing temporal dependence and the additional challenges of joint prediction across multiple horizons. We propose a kernel-based framework, called Time series Residual Analysis for Calibration with Kernels (TRACK), for estimating conditional quantiles of residual scores and constructing prediction sets from rolling-window forecasters. The framework accommodates heterogeneous uncertainty across horizons and accounts for dependence induced by rolling estimation, while allowing predictor memory and forecast horizons to grow with the sample size. Our finite-sample bounds quantify how temporal dependence and overlapping score windows affect calibration and coverage. We further derive bounds for conditional quantile estimation and pointwise conditional coverage discrepancy. The guarantees extend to more general predictors that admit an accurate rolling-window approximation. We also derive matching minimax bounds over classes of known score maps, clarifying how overlapping score windows affect calibration accuracy. Experimental results from simulations and applications to wind energy generation and bike sharing demand show that our methods improve regional conditional coverage while maintaining competitive interval widths.
Feature subsampling is used to accelerate gradient-boosted trees, yet standard approaches fix the feature count in advance and largely ignore the split-gain evidence accumulated during training. We introduce CRAFT-GBM (Confidence-Ranked Adaptive Feature Tracking), a native LightGBM extension that uses this evidence to adapt which---and, by default, how many---features are searched each boosting round. CRAFT-GBM transforms root-split gains into bounded observations and constructs joint time-uniform bounds valid under adaptive, non-stationary training histories. In its default endogenous mode, after initialization, CRAFT-GBM retains exactly the features that cannot yet be ruled out as best, letting the selected feature-set size $K_t$ emerge from training history rather than a pre-specified fraction. With probability at least $1-\delta$, every feature with the largest expected gain is retained jointly across all rounds after initialization. When a feature fraction $f$ is specified instead, CRAFT-GBM ranks features by their interval point estimates and provides an approximate top-$K$ guarantee for feature budget $K_f$. We implement CRAFT-GBM within LightGBM while preserving its native feature-bundling representation, and evaluate it across 14 public and synthetic dataset families and four noise-augmented settings. In the fully endogenous setting, CRAFT-GBM reduces per-fit training time by 39.74\% with 1.04\% lower test loss in the geometric mean across all 18 RQ1 experiments. Under pre-specified feature fraction $f$, Exogenous CRAFT-GBM also yields aggregate training-time reductions up to 13.52 percentage points larger than random feature subsampling, with no increase in aggregate test loss relative to full-feature LightGBM. Accumulated training evidence can thus support adaptive feature search with statistical guarantees while retaining the practical advantages of native gradient boosting.
We study the problem of learning directed acyclic graphs from observational data in nonlinear models and provide finite-sample, nonasymptotic statistical guarantees in these models. Although widely studied, statistical guarantees for gradient-based methods in nonlinear models are missing from the literature. Under a generalized linear model for each node, we analyze a global nonsmooth objective combining an exact acyclicity constraint with a quasi-minimax concave penalty, which can be optimized via standard gradient-based solvers. We establish finite-sample guarantees under this model, including the most widely used logistic and Poisson models. Experiments corroborate structure recovery accuracy and computational performance, including both standardized data and negative controls as baselines.
Gaussian processes form a popular and flexible class of prior distributions with principled uncertainty quantification, but the Gaussian likelihood on which the corresponding posterior relies makes them highly sensitive to outliers and contaminated observations. We study Gaussian process regression when among the $n$ observations, $m$ of these are possibly contaminated. A generic framework to obtain posterior contraction rates in this setting is considered, from a frequentist perspective as the sample size $n$ and the number of corrupted observations $m$ grow. We adopt a generalized Bayes approach in which the log--likelihood is replaced by a robustified version; we discuss various choices thereof, such as one recently proposed by Altamirano et al. that preserves conjugacy. We investigate the minimax rate of estimation under adversarial contamination in terms of the Sobolev smoothness $\beta$ of the regression function and $m$. The question of building adaptive procedures to both $\beta$ and $m$ is investigated, and we show that a polynomial loss for adaptation must be paid in general. We instantiate our results for Mat\'ern Gaussian process and deep ReLU network priors. Numerical experiments for existing and new robust algorithms are provided, illustrating the practical applicability of our results.
We study group-conditional conformal risk control (CRC) for disjoint and overlapping groups under bounded monotone losses. For disjoint groups, we establish exact finite-sample conditional risk control, two-sided bounds on the group-restricted risk, and fundamental limitations arising from local support and calibration-rank granularity. For overlapping groups, we study two exactly valid approaches: group-restricted calibration followed by union deployment, and direct calibration of the deployed prediction-set family. We characterize the union's limiting reconciliation gap, identify conditions for its disappearance, and establish sharp lower bounds for structured losses in specified product-form models. We also demonstrate a strict size advantage for direct calibration in an ordinal model. Experiments show that direct calibration can reduce prediction-set size, but its efficiency depends on group structure, loss type, and shape fitting. Results on historical diabetic-retinopathy scores further illustrate these tradeoffs. Overall, local support, reconciliation slack, and threshold heterogeneity determine efficiency, with neither approach uniformly dominating.
We study variational inference in the presence of distributional symmetry, focusing on the interaction between two distinct group actions on a sample space: a symmetry action that leaves a reference distribution invariant, and a transport action that generates a parametric variational family by pushforward. By analyzing the algebraic structure of the total transformation group generated by both actions and the induced adjoint representation of the symmetry group on the transport Lie algebra, we establish that the Riemannian gradient of the Kullback--Leibler divergence at the true transport parameter is constrained to lie in the trivial representation of the symmetry group. Consequently, symmetry forces the gradient to vanish along all non-trivial representations. Combining this representation-theoretic constraint with parameter identifiability yields an orthogonal decomposition of the transport Lie algebra into three interpretable subspaces: an identifiability kernel (along which the objective is locally constant), a bias subspace (where the gradient may be nonzero due to model misspecification), and a symmetry-breaking subspace (where the gradient is forced to vanish by symmetry alone). Finally, we characterize the residual bias directions in terms of distributional mismatch between the base model and the standardized target, and derive an explicit Stein-type moment-matching condition that eliminates this bias, extending the location-scale results of (Margossian and Saul, 2025) to general Lie group transformation families.
We study the minimax regret rate for episodic kernel-based reinforcement learning, where the expected reward and transition density belong to a Sobolev reproducing kernel Hilbert space (RKHS), with continuous state and action spaces, and adversarial initial states. For Sobolev smoothness parameter $\nu>0$, state dimension $d_S$, and action dimension $d_A$, we prove that the minimax expected regret over $T$ episodes is $\widetilde\Theta(T^{\max\{e_A,e_B\}})$, where $e_A=\frac{\nu+d_A}{2\nu+d_A}$, $e_B=\frac{\nu+d_A+3d_S/2}{2\nu+d_A+2d_S}$ with the other model parameters fixed. The two exponents reflect the distinct difficulties of action selection and learning across states. We introduce Sobolev Optimal Value Iteration (\texttt{SOVI}), which attains this rate from observed rewards and transitions. For fixed smoothness and reference-kernel lengthscale, its regret bound has a prefactor polynomial in the state-action dimension $d=d_S+d_A$, horizon $H$, reward and transition RKHS norm bounds $B_r,B_p$, and the norm-comparison factor $\Lambda$, up to logarithmic factors; $\Lambda=1$ for normalized Mat\'ern kernels. The algorithm separates initial action selection from continuation-value estimation and combines local polynomial approximation with the shared RKHS structure of the approximation errors. Our matching upper and lower bounds determine the optimal dependence on the number of episodes for Sobolev kernel-based reinforcement learning regret minimization. Thus, we partially address the open problem in Vakili(2024) under Sobolev kernel RL.
Learned barrier certificates provide a principled mechanism for establishing safety of stochastic systems, but their validity may deteriorate during deployment when the underlying dynamics differ from those used for training. We propose an anytime-valid runtime monitor that audits a learned barrier certificate using observed closed-loop trajectories. The monitor is intended to detect loss of statistical support for the deployed barrier condition; it does not by itself guarantee detection before unsafe entry or safety of a subsequent intervention. Specifically, the conditional barrier-drift condition is formulated as an episode-wise sequential hypothesis test, and observed barrier increments are transformed into a nonnegative test supermartingale, yielding finite-sample false-alarm control under arbitrary stopping times. Inspired by finite-horizon safety verification, we further introduce a deterministic temporal shaping of the monitoring boundary, making the monitor more conservative early in a prescribed deployment horizon while preserving anytime-validity. The monitor is designed to detect statistical evidence that the safety certificate is no longer consistent with the observed dynamics, rather than to predict unsafe entry. To assess the monitor empirically, we evaluate empirical alarm behavior under valid and invalid certificates. Code and data are available at https://github.com/FormalCert-AI/runtime-barrier-monitor.
Steinke and Zakynthinou(2020) introduces the Conditional Mutual Information (CMI) framework of analyzing the information complexity of learning algorithms based on algorithm-dependent information-theoretic quantities. We study one of these quantities, the evaluated Conditional Mutual Information (eCMI). It has been an interesting question whether the optimal PAC guarantee for VC classes can be recovered from the algorithm-dependent analyses via CMI. And we show that it is possible to recover this guarantee by constructing a learning algorithm whose eCMI is of order O(d) in the realizable case, where d is the VC-dimension of the concept class. Specially, our algorithm is a randomized Majority-of-5 base learners with optimal in-expectation generalization guarantee.
Knowledge graph learning offers a powerful framework for representing and inferring structured knowledge, with broad practical applications. Two obstacles limit current methods: labeled triples are scarce for each entity and relation, which makes expressive models hard to train, and scoring functions are designed ad hoc, which limits their generalizability and leaves them without theoretical grounding. We address both obstacles with a theoretically grounded framework, with end-to-end guarantees, that extends and subsumes existing methods. The framework proceeds in two stages: multi-view unsupervised pretraining that integrates heterogeneous sources of entity side information into a shared representation, followed by supervised learning across multiple relation types. We establish a unified nonasymptotic risk bound that combines the subspace recovery error of pretraining and the sample complexity of the labeled data in a single guarantee, quantifying formally how large-scale unlabeled data from multiple views benefits downstream knowledge prediction. Synthetic experiments validate each component of the theory, and experiments on large-scale real-world knowledge graph benchmarks confirm the effectiveness of our approach.
We prove the population-level identifiability of deep generative models (DGMs) with piecewise-affine (PWA) decoders and Gaussian mixture model (GMM) priors, in a purely unsupervised setting. We introduce three algebraic contrast principles for symmetry breaking: *domain contrast*, which trivializes the mixture symmetry group, a label-free counterpart of the sufficient-variability assumption of nonlinear ICA; *mechanism contrast*, which ensures every decoder branch gets an observational boundary across which it toggles alone; and *interaction contrast*, which forbids parameter conspiracies between latent components and decoder branches. Together they exploit the interplay between the discrete combinatorics of the PWA map and the continuous symmetry structure of the latent GMM. Continuity is replaced by algebraic symmetry conditions; injectivity is decoupled from structural identification and required only for pointwise inversion. Our results form a hierarchy: from *law identifiability* (LID; latent distribution up to a global affine map) through *map identifiability* (MID; decoder up to the same map) to *posterior* and *pointwise identifiability*. The ICA-form ambiguity emerges under conditions on diagonal component covariances. The interaction contrast is the only one of our contrasts required on the estimator side. To our knowledge this is the first to develop algebraic symmetry-breaking into an identifiability engine for a concrete nonlinear model class, the first to admit discontinuous decoders, and the first to handle fully non-injective decoders, where every observation admits multiple latent codes.
Digital Twins support real-time decision-making through predictive models of physical systems, yet their reliability can deteriorate as operating conditions evolve. Maintaining predictive fidelity under such changes is challenging, particularly when model adaptation relies on limited streaming data, and updated models must be evaluated before deployment. Here, we present an online adaptation framework that integrates gradient-based drift detection, parameter-efficient fine-tuning, and statistical model validation within model predictive control. Changes in prediction-loss gradients trigger batch-wise model updates using Low-Rank Adaptation, while a paired moving-block bootstrap test evaluates predictive improvement before deploying the updated model. The framework is evaluated using a stochastic dynamical system with abrupt changes and a simulated directed energy deposition process with gradual changes in material properties. Results show improvements in predictive accuracy and uncertainty estimation during the evaluated post-adaptation intervals, alongside improved reference tracking and reduced constraint violations in the manufacturing case study compared with the investigated baselines. These findings demonstrate the potential of integrating continual validation with model adaptation while acknowledging the trade-off between responsiveness and predictive reliability in evolving Digital Twins.
Many real-world processes can be represented as compositions of functions along a directed acyclic graph (DAG). In causal modelling, these correspond to the underlying mechanisms; in engineering, to multiple fidelity levels; and in gene-regulatory networks, to transcription factors. These functions are partially observed across the DAG, with noisy and heterogeneously sampled measurements, posing significant challenges for reconstruction, uncertainty propagation, and inference. To tackle these challenges, we place priors over functions and naturally arrive at Deep Gaussian Processes over DAGs. We theoretically study their prior-collapse behaviour and the effect of graph topology and intermediate observations on the preservation of information. We obtain almost-sure lower bounds on the asymptotic frequency of depths at which the distinction between inputs is preserved, identify broad kernel classes for which these hold, and prove an observation by \citep{dunlop2018} on the role of input connections. We offer a structured variational approximation that retains graph dependencies, preserves compositional uncertainty, and captures the explaining-away behaviour of colliders. Finally, we empirically validate our theory and methodology at scale and under DAG misspecification. We then model a latent-collider DAG, a protein signalling network, and a multi-fidelity \mbox{heavy-ion} collision emulation task, attaining state-of-the-art performance while recovering low-fidelity contributions and yielding interpretability over the simulator hierarchy.
We introduce a new tool, Express, for converting a non-causal attention approximation into a causal approximation with matching approximation guarantees. When combined with the state-of-the-art Thinformer approximation, Express improves upon the best known causal attention guarantees, delivering $\log^{3/2}(n)/s$ approximation error with only $O(s)$ memory and $O(s^2 \log^2(n))$ compression overhead for a sequence of length $n$. We pair these developments with an efficient I/O-aware Triton implementation, demonstrate substantial speedups over FlashAttention 2, and use Express to overcome four resource bottlenecks in the language modeling pipeline: long-context prefill, KV cache compression, long-form memory-constrained decoding, and long-form compute-constrained decoding.
Clustering is a central tool for discovering latent structure in unlabeled data; yet modern clustering pipelines often end with a hard assignment of each observation to a cluster without rigorous measures of assignment uncertainty. We propose a novel weighted conformal approach for constructing valid confidence sets for cluster labels. The key difficulty is that the labels available for calibration are not observed ground-truth labels, but synthetic labels produced by a data-dependent clustering algorithm. Our method develops a conformal inference algorithm that corrects the resulting mismatch with the latent target labels through weights by formulating conformal clustering as a conditional label-distribution shift problem. We first derive an oracle procedure that attains finite-sample marginal coverage and then develop a computationally tractable and implementable version using estimated conditional label probabilities and novel augmented calibration. We show that the coverage of the estimated-weight procedure depends on the estimator, giving an explicit bound on the loss relative to the nominal level. Empirical studies demonstrate that the proposed weighted approach offers improvements over the recently proposed split conformal clustering procedure in terms of informative confidence set size, especially in nonlinear and high-dimensional clustering applications.
Nearest-neighbor methods are fundamental to classical and modern machine learning, yet their geometric properties are typically analyzed under independent sampling. In this paper, we study the nearest-neighbor radii under dependent sampling. We consider strong mixing dependent observations and ask whether dependence changes the scale of nearest-neighbor neighborhoods. We establish distribution-free almost sure convergence under polynomial mixing and sharp non-asymptotic moment bounds under geometric mixing. The moment bounds depend on the local intrinsic dimension rather than the ambient dimension, making the results applicable to high-dimensional data supported on lower-dimensional manifolds. Synthetic experiments and real-world time-series benchmarks illustrate the theory, showing that nearest-neighbor geometry remains informative under dependent sampling.
Distributional causal inference requires estimating not only average treatment effects but also interventional outcome distributions, including quantiles, tail risks, and policy-dependent uncertainty. As a method for distributional causal inference, generative adversarial network (GAN)-based counterfactual methods are flexible tools for this task. However, these methods have several limitations. First, the objectives of certain techniques do not coincide with the statistical risk of the identifiable causal target, and therefore provide limited theoretical guarantees regarding estimable counterfactual distributions or optimality. Second, implementing target-design risks under observational sampling can require density-ratio estimation. In this paper, we propose GANICE (GAN for Interventional Conditional Estimation) with several advantages: it (i) specifies the conditional interventional distribution for each treatment--covariate state as the causal estimation target; (ii) estimates the conditional distribution such that its averaged Wasserstein risk is minimized; (iii) attains minimax optimality. GANICE achieves these advantages through the extended Wasserstein distance and cell-normalized estimation, with minimax bounds based on Wasserstein GAN oracle inequalities and an anisotropic Assouad construction; Besov theory characterizes the cellwise critics. Our experiments show that GANICE outperforms existing methods on most benchmarks.
\emph{Kullback-Leibler} (KL) regularization is ubiquitous in reinforcement learning algorithms in the form of \emph{reverse} or \emph{forward} KL. Recent studies have demonstrated $\epsilon^{-1}$-type fast rates for decision making under reverse KL regularization, in contrast to the standard $\epsilon^{-2}$-type sample complexity. However, for forward-KL-regularized objectives, existing statistical analyses are either not applicable or result in $\tilde{O}(\epsilon^{-2})$ slow rates. We take the first step towards addressing this problem via a streamlined analysis of forward-KL-regularized offline CBs. We give the first $\tilde{O}(\epsilon^{-1})$ upper bounds in tabular and general function approximation settings, both under notions of \emph{single-policy concentrability}. In particular, our convex-analytical pipeline unifies these settings by exploiting the pessimism principle in a novel way and completely bypasses the proof routines in previous works based on the mean value theorem, which might be of independent interest. Moreover, we provide rate-optimal lower bounds, manifesting the tightness of our upper bounds in terms of statistical rates. Our lower bounds also demonstrate that the forward-KL-regularized sample complexity recovers the unregularized slow rate in the low-regularization regime, similarly to the reverse-KL regularization.
We address the brittleness of Bayesian experimental design under model misspecification by formulating the problem as a max--min game between the experimenter and an adversarial nature subject to information-theoretic constraints. We demonstrate that this approach yields a robust objective governed by Sibson's $\alpha$-mutual information~(MI), which identifies the $\alpha$-tilted posterior as the robust belief update and establishes the R\'enyi divergence as the appropriate measure of conditional information gain. To mitigate the bias and variance of nested Monte Carlo estimators needed to estimate Sibson's $\alpha$-MI, we adopt a PAC-Bayes framework to search over stochastic design policies, yielding rigorous high-probability lower bounds on the robust expected information gain that explicitly control finite-sample error.