Stochastic differential equations (SDEs) are a fundamental tool for modelling dynamic processes, including gene regulatory networks (GRNs), contaminant transport, financial markets, and image generation. However, learning the underlying SDE from data is a challenging task, especially if individual trajectories are not observable. Motivated by burgeoning research in single-cell datasets, we present the first comprehensive approach for jointly identifying the drift and diffusion of an SDE from its temporal marginals. Assuming linear drift and additive diffusion, we show that non-identifiability can only arise if the initial distribution possesses generalized rotational symmetries. We further prove that even if this condition holds, the drift and diffusion can almost always be recovered from the marginals. Additionally, we show that the causal graph of any SDE with additive diffusion can be recovered from the identified SDE parameters. To complement this theory, we adapt entropy-regularized optimal transport to handle anisotropic diffusion and introduce APPEX (Alternating Projection Parameter Estimation from $X_0$), an iterative algorithm designed to estimate the drift, diffusion, and causal graph of an additive noise SDE, solely from temporal marginals, and demonstrate its effectiveness on simulated data from linear additive noise SDEs.
This dissertation develops and justifies a novel method for deriving approximate formulas to estimate two parameters in stochastic volatility diffusion models with exponentially-affine characteristic functions and single- or two-factor variance. These formulas aim to improve the accuracy of option pricing and enhance the calibration process by providing reliable initial values for local minimization algorithms. The parameters relate to the volatility of the stochastic factor in instantaneous variance dynamics and the correlation between stochastic factors and asset price dynamics. The study comprises five chapters. Chapter one outlines the currency option market, pricing methods, and the general structure of stochastic volatility models. Chapter two derives the replication strategy dynamics and introduces a new two-factor volatility model: the OUOU model. Chapter three analyzes the distribution and surface dynamics of implied volatilities using principal component and common factor analysis. Chapter four discusses calibration methods for stochastic volatility models, particularly the Heston model, and presents the new Implied Central Moments method to estimate parameters in the Heston and Sch\"obel-Zhu models. Extensions to two-factor models, Bates and OUOU, are also explored. Chapter five evaluates the performance of the proposed formulas on the EURUSD options market, demonstrating the superior accuracy of the new method. The dissertation successfully meets its research objectives, expanding tools for derivative pricing and risk assessment. Key contributions include faster and more precise parameter estimation formulas and the introduction of the OUOU model - an extension of the Sch\"obel-Zhu model with a semi-analytical valuation formula for European options, previously unexamined in the literature.
Stochastic Petri Nets (SPNs) are an increasingly popular tool of choice for modeling discrete-event dynamics in areas such as epidemiology and systems biology, yet their parameter estimation remains challenging in general and in particular when transition rates depend on external covariates and explicit likelihoods are unavailable. We introduce a neural-surrogate (neural-network--based approximation of the posterior distribution) framework that predicts the coefficients of known covariate-dependent rate functions directly from noisy, partially observed token trajectories. Our model employs a lightweight 1D Convolutional Residual Network trained end-to-end on Gillespie-simulated SPN realizations, learning to invert system dynamics under realistic conditions of event dropout. During inference, Monte Carlo dropout provides calibrated uncertainty bounds together with point estimates. On synthetic SPNs with 20% missing events, our surrogate recovers rate-function coefficients with an RMSE = 0.108 and substantially runs faster than traditional Bayesian approaches. These results demonstrate that data-driven, likelihood-free surrogates can enable accurate, robust, and real-time parameter recovery in complex, partially observed discrete-event systems.
In epidemic modeling, outliers can distort parameter estimation and ultimately lead to misguided public health decisions. Although there are existing robust methods that can mitigate this distortion, the ability to simultaneously detect outliers is equally vital for identifying potential disease hotspots. In this work, we introduce a robust spatiotemporal generalized additive model (RST-GAM) to address this need. We accomplish this with a mean-shift parameter to quantify and adjust for the effects of outliers and rely on adaptive Lasso regularization to model the sparsity of outlying observations. We use univariate polynomial splines and bivariate penalized splines over triangulations to estimate the functional forms and a data-thinning approach for data-adaptive weight construction. We derive a scalable proximal algorithm to estimate model parameters by minimizing a convex negative log-quasi-likelihood function. Our algorithm uses adaptive step-sizes to ensure global convergence of the resulting iterate sequence. We establish error bounds and selection consistency for the estimated parameters and demonstrate our model's effectiveness through numerical studies under various outlier scenarios. Finally, we demonstrate the practical utility of RST-GAM by analyzing county-level COVID-19 infection data in the United States, highlighting its potential to inform public health decision-making.
Elucidating the biomechanical behavior of the myocardium is crucial for understanding cardiac physiology, but cannot be directly inferred from clinical imaging and typically requires finite element (FE) simulations. However, conventional FE methods are computationally expensive and often fail to reproduce observed cardiac motions. We propose IMC-PINN-FE, a physics-informed neural network (PINN) framework that integrates imaged motion consistency (IMC) with FE modeling for patient-specific left ventricular (LV) biomechanics. Cardiac motion is first estimated from MRI or echocardiography using either a pre-trained attention-based network or an unsupervised cyclic-regularized network, followed by extraction of motion modes. IMC-PINN-FE then rapidly estimates myocardial stiffness and active tension by fitting clinical pressure measurements, accelerating computation from hours to seconds compared to traditional inverse FE. Based on these parameters, it performs FE modeling across the cardiac cycle at 75x speedup. Through motion constraints, it matches imaged displacements more accurately, improving average Dice from 0.849 to 0.927, while preserving realistic pressure-volume behavior. IMC-PINN-FE advances previous PINN-FE models by introducing back-computation of material properties and better motion fidelity. Using motion from a single subject to reconstruct shape modes also avoids the need for large datasets and improves patient specificity. IMC-PINN-FE offers a robust and efficient approach for rapid, personalized, and image-consistent cardiac biomechanical modeling.
Deep neural networks have been shown to achieve exceptional performance for computer vision tasks like image recognition, segmentation, and reconstruction or denoising. Here, we evaluate the ultimate performance limits of deep convolutional neural network models for image reconstruction, by comparing them against the standard quantum limit set by shot-noise and the Heisenberg limit on precision. We train U-Net models on images of natural objects illuminated with coherent states of light, and find that the average mean-squared error of the reconstructions can surpass the standard quantum limit, and in some cases reaches the Heisenberg limit. Further, we train models on well-parameterized images for which we can calculate the quantum Cram\'er-Rao bound to determine the minimum possible measurable variance of an estimated parameter for a given probe state. We find the mean-squared error of the model predictions reaches these bounds calculated for the parameters, across a variety of parameterized images. These results suggest that deep convolutional neural networks can learn to become the optimal estimators allowed by the laws of physics, performing parameter estimation and image reconstruction at the ultimate possible limits of precision for the case of classical illumination of the object.
There is growing interest in extending average treatment effect (ATE) estimation to incorporate non-tabular data, such as images and text, which may act as sources of confounding. Neglecting these effects risks biased results and flawed scientific conclusions. However, incorporating non-tabular data necessitates sophisticated feature extractors, often in combination with ideas of transfer learning. In this work, we investigate how latent features from pre-trained neural networks can be leveraged to adjust for sources of confounding. We formalize conditions under which these latent features enable valid adjustment and statistical inference in ATE estimation, demonstrating results along the example of double machine learning. We discuss critical challenges inherent to latent feature learning and downstream parameter estimation arising from the high dimensionality and non-identifiability of representations. Common structural assumptions for obtaining fast convergence rates with additive or sparse linear models are shown to be unrealistic for latent features. We argue, however, that neural networks are largely insensitive to these issues. In particular, we show that neural networks can achieve fast convergence rates by adapting to intrinsic notions of sparsity and dimension of the learning problem.
We introduce an approach to topic modelling with document-level covariates that remains tractable in the face of large text corpora. This is achieved by de-emphasizing the role of parameter estimation in an underlying probabilistic model, assuming instead that the data come from a fixed but unknown distribution whose statistical functionals are of interest. We propose combining a convex formulation of non-negative matrix factorization with standard regression techniques as a fast-to-compute and useful estimate of such a functional. Uncertainty quantification can then be achieved by reposing non-parametric resampling methods on top of this scheme. This is in contrast to popular topic modelling paradigms, which posit a complex and often hard-to-fit generative model of the data. We argue that the simple, non-parametric approach advocated here is faster, more interpretable, and enjoys better inferential justification than said generative models. Finally, our methods are demonstrated with an application analysing covariate effects on discourse of flavours attributed to Canadian beers.
Physics-inspired inference often hinges on the ability to construct a likelihood, or the probability of observing a sequence of data given a model. These likelihoods can be directly maximized for parameter estimation, incorporated into Bayesian frameworks, or even used as loss functions in neural networks. Yet, many models, despite being conceptually simple, lack tractable likelihoods. A notable example arises in estimating protein production from snapshot measurements of actively dividing cells. Here, the challenge stems from cell divisions occurring at non-Exponentially distributed intervals with each division stochastically partitioning protein content between daughter cells, making protein counts in any given cell a function of its full division history. Such history dependence precludes a straightforward likelihood based on a (standard Markovian) master equation. Instead, we employ conditional normalizing flows (a class of neural network models designed to learn probability distributions) to approximate otherwise intractable likelihoods from simulated data. As a case study, we examine activation of the \emph{glc3} gene in yeast involved in glycogen synthesis and expressed under nutrient-limiting conditions. We monitor this activity using snapshot fluorescence measurements via flow cytometry, where GFP expression reflects \emph{glc3} promoter activity. A na\"ive analysis of flow cytometry data ignoring cell division suggests many cells are active with low expression. However, fluorescent proteins persist and can be inherited, so cells may appear active from retaining ancestral fluorescence. Explicitly accounting for the (non-Markovian) effects of cell division reveals \emph{glc3} is mostly inactive under stress, showing that while cells occasionally activate it, expression is brief and transient.
Statistical inference in physics is often based on samples from a generator (sometimes referred to as a ``forward model") that emulate experimental data and depend on parameters of the underlying theory. Modern machine learning has supercharged this workflow to enable high-dimensional and unbinned analyses to utilize much more information than ever before. We propose a general framework for describing the integration of machine learning with generators called Generator Based Inference (GBI). A well-studied special case of this setup is Simulation Based Inference (SBI) where the generator is a physics-based simulator. In this work, we examine other methods within the GBI toolkit that use data-driven methods to build the generator. In particular, we focus on resonant anomaly detection, where the generator describing the background is learned from sidebands. We show how to perform machine learning-based parameter estimation in this context with data-derived generators. This transforms the statistical outputs of anomaly detection to be directly interpretable and the performance on the LHCO community benchmark dataset establishes a new state-of-the-art for anomaly detection sensitivity.
Inverse problem or parameter estimation of ordinary differential equations is a process of obtaining the best parameters using experimental measurements of the states. Single (Multiple)-shooting is a type of sequential optimization method that minimizes the error in the measured and numerically integrated states. However, this requires computing sensitivities i.e. the derivatives of states with respect to the parameters over the numerical integrator, which can get computationally expensive. To address this challenge, many interpolation-based approaches have been proposed to either reduce the computational cost of sensitivity calculations or eliminate their need. In this paper, we use a bi-level optimization framework that leverages interpolation and exploits the structure of the differential equation to solve an inner convex optimization problem. We apply this method to two different problem formulations. First, parameter estimation for differential equations, and delayed differential equations, where the model structure is known but the parameters are unknown. Second, model discovery problems, where both the model structure and parameters are unknown.
Recent advances in language modeling have underscored the role of preference feedback in enhancing model performance. This paper investigates the conditions under which preference feedback improves parameter estimation in classes of continuous parametric distributions. In our framework, the learner observes pairs of samples from an unknown distribution along with their relative preferences depending on the same unknown parameter. We show that preference-based M-estimators achieve a better asymptotic variance than sample-only M-estimators, further improved by deterministic preferences. Leveraging the hard constraints revealed by deterministic preferences, we propose an estimator achieving an estimation error scaling of $\mathcal{O}(1/n)$ -- a significant improvement over the $\Theta(1/\sqrt{n})$ rate attainable with samples alone. Next, we establish a lower bound that matches this accelerated rate; up to dimension and problem-dependent constants. While the assumptions underpinning our analysis are restrictive, they are satisfied by notable cases such as Gaussian or Laplace distributions for preferences based on the log-probability reward.
Bayesian network classifiers (BNCs) possess a number of properties desirable for a modern classifier: They are easily interpretable, highly scalable, and offer adaptable complexity. However, traditional methods for learning BNCs have historically underperformed when compared to leading classification methods such as random forests. Recent parameter smoothing techniques using hierarchical Dirichlet processes (HDPs) have enabled BNCs to achieve performance competitive with random forests on categorical data, but these techniques are relatively inflexible, and require a complicated, specialized sampling process. In this paper, we introduce a novel method for parameter estimation that uses a log-linear regression to approximate the behaviour of HDPs. As a linear model, our method is remarkably flexible and simple to interpret, and can leverage the vast literature on learning linear models. Our experiments show that our method can outperform HDP smoothing while being orders of magnitude faster, remaining competitive with random forests on categorical data.
To be fully useful for public health practice, models for epidemic response must be able to do more than predict -- it is also important to incorporate the mechanisms underlying transmission dynamics to enable policymakers and practitioners to be able to evaluate what-if scenarios and intervention options. However, most mechanistic models suffer from uncertainty in both the parameters (e.g., parameter unidentifiability) and the model structure itself, which can hinder both successful parameter estimation and model interpretation. To enable rapid development of interpretable and parsimonious mechanistic models, we use penalized regression and covariate selection methods to integrate parameter identifiability and model selection directly into the parameter estimation procedure for (in this case) traditional ordinary differential equation (ODE) models. For both simulated and real-world epidemiological data, we demonstrate that the LASSO-ODE framework is highly effective in selecting a parsimonious, identifiable model from larger, more realistic but potentially unidentifiable models, from realistically sparse data with only a single measured compartment and multiple latent (unobserved) variables. While we focus on epidemic models in this paper as a case study, these same approaches are applicable to a wide range of application areas that are faced with relatively sparse data but a need for realistic mechanistic models (e.g. mathematical oncology and mathematical biology more broadly). Additionally, the cross-validation techniques designed for time series data introduced in our study can be used across a range of time series analysis and modeling approaches.
Hollmann et al. (Nature 637 (2025) 319-326) recently introduced TabPFN, a transformer-based deep learning model for regression and classification on tabular data, which they claim "outperforms all previous methods on datasets with up to 10,000 samples by a wide margin, using substantially less training time." Furthermore, they have called TabPFN a "foundation model" for tabular data, as it can support "data generation, density estimation, learning reusable embeddings and fine-tuning". If these statements are well-supported, TabPFN may have the potential to supersede existing modeling approaches on a wide range of statistical tasks, mirroring a similar revolution in other areas of artificial intelligence that began with the advent of large language models. In this paper, we provide a tailored explanation of how TabPFN works for a statistics audience, by emphasizing its interpretation as approximate Bayesian inference. We also provide more evidence of TabPFN's "foundation model" capabilities: We show that an out-of-the-box application of TabPFN vastly outperforms specialized state-of-the-art methods for semi-supervised parameter estimation, prediction under covariate shift, and heterogeneous treatment effect estimation. We further show that TabPFN can outperform LASSO at sparse regression and can break a robustness-efficiency trade-off in classification. All experiments can be reproduced using the code provided at https://github.com/qinglong-tian/tabpfn_study (https://github.com/qinglong-tian/tabpfn_study).
Structural identifiability -- the theoretical ability to uniquely recover model parameters from ideal, noise-free data -- is a prerequisite for reliable parameter estimation in epidemic modeling. Despite its importance, structural identifiability analysis remains underutilized in the infectious disease modeling literature. In this tutorial, we present a practical and reproducible workflow for conducting structural identifiability analysis of ordinary differential equation models using the Julia package StructuralIdentifiability.jl. We apply the tool to a range of epidemic models, including SEIR variants with asymptomatic and pre-symptomatic transmission, vector-borne systems, and models incorporating hospitalization and disease-induced mortality. We compare results from StructuralIdentifiability.jl with those obtained using DAISY, a widely used differential algebra tool, and highlight cases where the Julia package succeeds in analyzing models that DAISY cannot handle. In particular, StructuralIdentifiability.jl efficiently handles high-dimensional systems and provides symbolic checks for assumptions often overlooked in other methods. Our findings underscore how identifiability depends on model structure, the availability of initial conditions, and the choice of observed states. All code and diagrams are publicly available, making this tutorial a valuable reference for researchers and educators working with dynamic disease models.
In many scientific and industrial applications, we are given a handful of instances (a 'small ensemble') of a spatially distributed quantity (a 'field') but would like to acquire many more. For example, a large ensemble of global temperature sensitivity fields from a climate model can help farmers, insurers, and governments plan appropriately. When acquiring more data is prohibitively expensive -- as is the case with climate models -- statistical emulation offers an efficient alternative for simulating synthetic yet realistic fields. However, parameter inference using maximum likelihood estimation (MLE) is computationally prohibitive, especially for large, non-stationary fields. Thus, many recent works train neural networks to estimate parameters given spatial fields as input, sidestepping MLE completely. In this work we focus on a popular class of parametric, spatially autoregressive (SAR) models. We make a simple yet impactful observation; because the SAR parameters can be arranged on a regular grid, both inputs (spatial fields) and outputs (model parameters) can be viewed as images. Using this insight, we demonstrate that image-to-image (I2I) networks enable faster and more accurate parameter estimation for a class of non-stationary SAR models with unprecedented complexity.
The recovery of block-sparse signals with unknown structural patterns remains a fundamental challenge in structured sparse signal reconstruction. By proposing a variance transformation framework, this paper unifies existing pattern-based block sparse Bayesian learning methods, and introduces a novel space power prior based on undirected graph models to adaptively capture the unknown patterns of block-sparse signals. By combining the EM algorithm with high-order equation root-solving, we develop a new structured sparse Bayesian learning method, SPP-SBL, which effectively addresses the open problem of space coupling parameter estimation in pattern-based methods. We further demonstrate that learning the relative values of space coupling parameters is key to capturing unknown block-sparse patterns and improving recovery accuracy. Experiments validate that SPP-SBL successfully recovers various challenging structured sparse signals (e.g., chain-structured signals and multi-pattern sparse signals) and real-world multi-modal structured sparse signals (images, audio), showing significant advantages in recovery accuracy across multiple metrics.
Physics-Informed Neural Networks (PINNs) have been widely used to obtain solutions to various physical phenomena modeled as Differential Equations. As PINNs are not naturally equipped with mechanisms for Uncertainty Quantification, some work has been done to quantify the different uncertainties that arise when dealing with PINNs. In this paper, we use a two-step procedure to train Bayesian Neural Networks that provide uncertainties over the solutions to differential equation systems provided by PINNs. We use available error bounds over PINNs to formulate a heteroscedastic variance that improves the uncertainty estimation. Furthermore, we solve forward problems and utilize the obtained uncertainties when doing parameter estimation in inverse problems in cosmology.
As Large Language Models (LLMs) and other AI systems evolve, robustly estimating their capabilities from inherently stochastic outputs while systematically quantifying uncertainty in these estimates becomes increasingly important. Further, advanced AI evaluations often have a nested hierarchical structure, exhibit high levels of complexity, and come with high costs in testing the most advanced AI systems. To address these challenges, we introduce HiBayES, a generalizable Hierarchical Bayesian modeling framework for AI Evaluation Statistics. HiBayES supports robust inferences in classical question-answer benchmarks and advanced agentic evaluations, particularly in low-data scenarios (e.g., < 20 data points per evaluation). Built on Generalized Linear Models (GLMs), Bayesian data analysis, and formal model comparison, HiBayES provides principled uncertainty quantification and robust parameter estimation. This paper offers a comprehensive introduction to HiBayES, including illustrative examples, comparisons to conventional statistical methods, and practical guidance for implementing multilevel Bayesian GLMs. Additionally, we provide a HiBayES software package [4] (Beta version) for out-of-the-box implementation.