Recently, Neural Ordinary Differential Equations has emerged as a powerful framework for modeling physical simulations without explicitly defining the ODEs governing the system, but instead learning them via machine learning. However, the question: "Can Bayesian learning frameworks be integrated with Neural ODE's to robustly quantify the uncertainty in the weights of a Neural ODE?" remains unanswered. In an effort to address this question, we primarily evaluate the following categories of inference methods: (a) The No-U-Turn MCMC sampler (NUTS), (b) Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) and (c) Stochastic Langevin Gradient Descent (SGLD). We demonstrate the successful integration of Neural ODEs with the above Bayesian inference frameworks on classical physical systems, as well as on standard machine learning datasets like MNIST, using GPU acceleration. On the MNIST dataset, we achieve a posterior sample accuracy of 98.5% on the test ensemble of 10,000 images. Subsequently, for the first time, we demonstrate the successful integration of variational inference with normalizing flows and Neural ODEs, leading to a powerful Bayesian Neural ODE object. Finally, considering a predator-prey model and an epidemiological system, we demonstrate the probabilistic identification of model specification in partially-described dynamical systems using universal ordinary differential equations. Together, this gives a scientific machine learning tool for probabilistic estimation of epistemic uncertainties.
In the vanishing learning rate regime, stochastic gradient descent (SGD) is now relatively well understood. In this work, we propose to study the basic properties of SGD and its variants in the non-vanishing learning rate regime. The focus is on deriving exactly solvable results and discussing their implications. The main contributions of this work are to derive the stationary distribution for discrete-time SGD in a quadratic loss function with and without momentum; in particular, one implication of our result is that the fluctuation caused by discrete-time dynamics takes a distorted shape and is dramatically larger than a continuous-time theory could predict. Examples of applications of the proposed theory considered in this work include the approximation error of variants of SGD, the effect of minibatch noise, the optimal Bayesian inference, the escape rate from a sharp minimum, and the stationary covariance of a few second-order methods including damped Newton's method, natural gradient descent, and Adam.
Latent Factor Model (LFM) is one of the most successful methods for Collaborative filtering (CF) in the recommendation system, in which both users and items are projected into a joint latent factor space. Base on matrix factorization applied usually in pattern recognition, LFM models user-item interactions as inner products of factor vectors of user and item in that space and can be efficiently solved by least square methods with optimal estimation. However, such optimal estimation methods are prone to overfitting due to the extreme sparsity of user-item interactions. In this paper, we propose a Bayesian treatment for LFM, named Bayesian Latent Factor Model (BLFM). Based on observed user-item interactions, we build a probabilistic factor model in which the regularization is introduced via placing prior constraint on latent factors, and the likelihood function is established over observations and parameters. Then we draw samples of latent factors from the posterior distribution with Variational Inference (VI) to predict expected value. We further make an extension to BLFM, called BLFMBias, incorporating user-dependent and item-dependent biases into the model for enhancing performance. Extensive experiments on the movie rating dataset show the effectiveness of our proposed models by compared with several strong baselines.
Bayesian neural networks (BNNs) have been long considered an ideal, yet unscalable solution for improving the robustness and the predictive uncertainty of deep neural networks. While they could capture more accurately the posterior distribution of the network parameters, most BNN approaches are either limited to small networks or rely on constraining assumptions such as parameter independence. These drawbacks have enabled prominence of simple, but computationally heavy approaches such as Deep Ensembles, whose training and testing costs increase linearly with the number of networks. In this work we aim for efficient deep BNNs amenable to complex computer vision architectures, e.g. ResNet50 DeepLabV3+, and tasks, e.g. semantic segmentation, with fewer assumptions on the parameters. We achieve this by leveraging variational autoencoders (VAEs) to learn the interaction and the latent distribution of the parameters at each network layer. Our approach, Latent-Posterior BNN (LP-BNN), is compatible with the recent BatchEnsemble method, leading to highly efficient ({in terms of computation and} memory during both training and testing) ensembles. LP-BNN s attain competitive results across multiple metrics in several challenging benchmarks for image classification, semantic segmentation and out-of-distribution detection.
We propose two generic methods for improving semi-supervised learning (SSL). The first integrates weight perturbation (WP) into existing "consistency regularization" (CR) based methods. We implement WP by leveraging variational Bayesian inference (VBI). The second method proposes a novel consistency loss called "maximum uncertainty regularization" (MUR). While most consistency losses act on perturbations in the vicinity of each data point, MUR actively searches for "virtual" points situated beyond this region that cause the most uncertain class predictions. This allows MUR to impose smoothness on a wider area in the input-output manifold. Our experiments show clear improvements in classification errors of various CR based methods when they are combined with VBI or MUR or both.
Many probabilistic inference problems such as stochastic filtering or the computation of rare event probabilities require model analysis under initial and terminal constraints. We propose a solution to this bridging problem for the widely used class of population-structured Markov jump processes. The method is based on a state-space lumping scheme that aggregates states in a grid structure. The resulting approximate bridging distribution is used to iteratively refine relevant and truncate irrelevant parts of the state-space. This way the algorithm learns a well-justified finite-state projection yielding guaranteed lower bounds for the system behavior under endpoint constraints. We demonstrate the method's applicability to a wide range of problems such as Bayesian inference and the analysis of rare events.
Mathematical modelling has successfully been used to provide quantitative descriptions of many viral infections, but for the Ebola virus, which requires biosafety level 4 facilities for experimentation, modelling can play a crucial role. Ebola modelling efforts have primarily focused on in vivo virus kinetics, e.g., in animal models, to aid the development of antivirals and vaccines. But, thus far, these studies have not yielded a detailed specification of the infection cycle, which could provide a foundational description of the virus kinetics and thus a deeper understanding of their clinical manifestation. Here, we obtain a diverse experimental data set of the Ebola infection in vitro, and then make use of Bayesian inference methods to fully identify parameters in a mathematical model of the infection. Our results provide insights into the distribution of time an infected cell spends in the eclipse phase (the period between infection and the start of virus production), as well as the rate at which infectious virions lose infectivity. We suggest how these results can be used in future models to describe co-infection with defective interfering particles, which are an emerging alternative therapeutic.
Multi-Delay single-shot arterial spin labeling (ASL) imaging provides accurate cerebral blood flow (CBF) and, in addition, arterial transit time (ATT) maps but the inherent low SNR can be challenging. Especially standard fitting using non-linear least squares often fails in regions with poor SNR, resulting in noisy estimates of the quantitative maps. State-of-the-art fitting techniques improve the SNR by incorporating prior knowledge in the estimation process which typically leads to spatial blurring. To this end, we propose a new estimation method with a joint spatial total generalized variation regularization on CBF and ATT. This joint regularization approach utilizes shared spatial features across maps to enhance sharpness and simultaneously improves noise suppression in the final estimates. The proposed method is evaluated at three levels, first on synthetic phantom data including pathologies, followed by in vivo acquisitions of healthy volunteers, and finally on patient data following an ischemic stroke. The quantitative estimates are compared to two reference methods, non-linear least squares fitting and a state-of-the-art ASL quantification algorithm based on Bayesian inference. The proposed joint regularization approach outperforms the reference implementations, substantially increasing the SNR in CBF and ATT while maintaining sharpness and quantitative accuracy in the estimates.
Recent research finds that forecasting electricity prices is very relevant. In many applications, it might be interesting to predict daily electricity prices by using their own lags or renewable energy sources. However, the recent turmoil of energy prices and the Russian-Ukrainian war increased attention in evaluating the relevance of industrial production and the Purchasing Managers' Index output survey in forecasting the daily electricity prices. We develop a Bayesian reverse unrestricted MIDAS model which accounts for the mismatch in frequency between the daily prices and the monthly macro variables in Germany and Italy. We find that the inclusion of macroeconomic low frequency variables is more important for short than medium term horizons by means of point and density measures. In particular, accuracy increases by combining hard and soft information, while using only surveys gives less accurate forecasts than using only industrial production data.
A Hidden Markov Model for intraday momentum trading is presented which specifies a latent momentum state responsible for generating the observed securities' noisy returns. Existing momentum trading models suffer from time-lagging caused by the delayed frequency response of digital filters. Time-lagging results in a momentum signal of the wrong sign, when the market changes trend direction. A key feature of this state space formulation, is no such lagging occurs, allowing for accurate shifts in signal sign at market change points. The number of latent states in the model is estimated using three techniques, cross validation, penalized likelihood criteria and simulation-based model selection for the marginal likelihood. All three techniques suggest either 2 or 3 hidden states. Model parameters are then found using Baum-Welch and Markov Chain Monte Carlo, whilst assuming a single (discretized) univariate Gaussian distribution for the emission matrix. Often a momentum trader will want to condition their trading signals on additional information. To reflect this, learning is also carried out in the presence of side information. Two sets of side information are considered, namely a ratio of realized volatilities and intraday seasonality. It is shown that splines can be used to capture statistically significant relationships from this information, allowing returns to be predicted. An Input Output Hidden Markov Model is used to incorporate these univariate predictive signals into the transition matrix, presenting a possible solution for dealing with the signal combination problem. Bayesian inference is then carried out to predict the securities $t+1$ return using the forward algorithm. Simple modifications to the current framework allow for a fully non-parametric model with asynchronous prediction.
This paper proposes that the thalamus is the site of a wave excitation, whose function is to represent the locations of things around the animal. Neurons couple to the wave as transmitters and receivers. The wave acts as an analogue representation of local space. This has benefits over a purely neural representation of space. Several lines of evidence support this hypothesis; both theoretical, concerning efficient Bayesian inference in the brain, and empirical, concerning the neuro-anatomy of the thalamus. Across all species, the most basic function of the brain is to coordinate movements in space. To represent positions in space only by neural firing rates would be complex and inefficient. It is possible that that the brain represents 3D space in a direct and natural way, by a 3D wave
Linear Quadratic Gaussian (LQG) control is a framework first introduced in control theory that provides an optimal solution to linear problems of regulation in the presence of uncertainty. This framework combines Kalman-Bucy filters for the estimation of hidden states with Linear Quadratic Regulators for the control of their dynamics. Nowadays, LQG is also a common paradigm in neuroscience, where it is used to characterise different approaches to sensorimotor control based on state estimators, forward and inverse models. According to this paradigm, perception can be seen as a process of Bayesian inference and action as a process of optimal control. Recently, active inference has been introduced as a process theory derived from a variational approximation of Bayesian inference problems that describes, among others, perception and action in terms of (variational and expected) free energy minimisation. Active inference relies on a mathematical formalism similar to LQG, but offers a rather different perspective on problems of sensorimotor control in biological systems based on a process of biased perception. In this note we compare the mathematical treatments of these two frameworks for linear systems, focusing on their respective assumptions and highlighting their commonalities and technical differences.
Adaptive behavior emerges through a dynamic interaction between cognitive agents and changing environmental demands. The investigation of information processing underlying adaptive behavior relies on controlled experimental settings in which individuals are asked to accomplish demanding tasks whereby a hidden state or an abstract rule has to be learned dynamically. Although performance in such tasks is regularly considered as a proxy for measuring high-level cognitive processes, the standard approach consists in summarizing response patterns by simple heuristic scoring measures. With this work, we propose and validate a new computational Bayesian model accounting for individual performance in the established Wisconsin Card Sorting Test. We embed the new model within the mathematical framework of Bayesian Brain Theory, according to which beliefs about the hidden environmental states are dynamically updated following the logic of Bayesian inference. Our computational model maps distinct cognitive processes into separable, neurobiologically plausible, information-theoretic constructs underlying observed response patterns. We assess model identification and expressiveness in accounting for meaningful human performance through extensive simulation studies. We further apply the model to real behavioral data in order to highlight the utility of the proposed model in recovering cognitive dynamics at an individual level. Practical and theoretical implications of our computational modeling approach for clinical and cognitive neuroscience research are finally discussed, as well as potential future improvements.
Contribution of this paper lies in the formulation and estimation of a generalized model for stochastic frontier analysis (SFA) that nests virtually all forms used and includes some that have not been considered so far. The model is based on the generalized t distribution for the observation error and the generalized beta distribution of the second kind for the inefficiency-related term. We use this general error structure framework for formal testing, to compare alternative specifications and to conduct model averaging. This allows us to deal with model specification uncertainty, which is one of the main unresolved issues in SFA, and to relax a number of potentially restrictive assumptions embedded within existing SF models. We also develop Bayesian inference methods that are less restrictive compared to the ones used so far and demonstrate feasible approximate alternatives based on maximum likelihood.
Researchers increasingly wish to estimate time-varying parameter (TVP) regressions which involve a large number of explanatory variables. Including prior information to mitigate over-parameterization concerns has led to many using Bayesian methods. However, Bayesian Markov Chain Monte Carlo (MCMC) methods can be very computationally demanding. In this paper, we develop computationally efficient Bayesian methods for estimating TVP models using an integrated rotated Gaussian approximation (IRGA). This exploits the fact that whereas constant coefficients on regressors are often important, most of the TVPs are often unimportant. Since Gaussian distributions are invariant to rotations we can split the the posterior into two parts: one involving the constant coefficients, the other involving the TVPs. Approximate methods are used on the latter and, conditional on these, the former are estimated with precision using MCMC methods. In empirical exercises involving artificial data and a large macroeconomic data set, we show the accuracy and computational benefits of IRGA methods.
Many living and complex systems exhibit second order emergent dynamics. Limited experimental access to the configurational degrees of freedom results in data that appears to be generated by a non-Markovian process. This poses a challenge in the quantitative reconstruction of the model from experimental data, even in the simple case of equilibrium Langevin dynamics of Hamiltonian systems. We develop a novel Bayesian inference approach to learn the parameters of such stochastic effective models from discrete finite length trajectories. We first discuss the failure of naive inference approaches based on the estimation of derivatives through finite differences, regardless of the time resolution and the length of the sampled trajectories. We then derive, adopting higher order discretization schemes, maximum likelihood estimators for the model parameters that provide excellent results even with moderately long trajectories. We apply our method to second order models of collective motion and show that our results also hold in the presence of interactions.
Biomedical data, particularly in the field of genomics, has characteristics which make it challenging for machine learning applications - it can be sparse, high dimensional and noisy. Biomedical applications also present challenges to model selection - whilst powerful, accurate predictions are necessary, they alone are not sufficient for a model to be deemed useful. Due to the nature of the predictions, a model must also be trustworthy and transparent, empowering a practitioner with confidence that its use is appropriate and reliable. In this paper, we propose that this can be achieved through the use of judiciously built feature sets coupled with Bayesian models, specifically Gaussian processes. We apply Gaussian processes to drug discovery, using inexpensive transcriptomic profiles from human cell lines to predict animal kidney and liver toxicity after treatment with specific chemical compounds. This approach has the potential to reduce invasive and expensive animal testing during clinical trials if in vitro human cell line analysis can accurately predict model animal phenotypes. We compare results across a range of feature sets and models, to highlight model importance for medical applications.
In this paper, we write the time-varying parameter (TVP) regression model involving K explanatory variables and T observations as a constant coefficient regression model with KT explanatory variables. In contrast with much of the existing literature which assumes coefficients to evolve according to a random walk, a hierarchical mixture model on the TVPs is introduced. The resulting model closely mimics a random coefficients specification which groups the TVPs into several regimes. These flexible mixtures allow for TVPs that feature a small, moderate or large number of structural breaks. We develop computationally efficient Bayesian econometric methods based on the singular value decomposition of the KT regressors. In artificial data, we find our methods to be accurate and much faster than standard approaches in terms of computation time. In an empirical exercise involving inflation forecasting using a large number of predictors, we find our models to forecast better than alternative approaches and document different patterns of parameter change than are found with approaches which assume random walk evolution of parameters.
Fourier-like summation of several grid cell modules with different spatial frequencies in the medial entorhinal cortex (MEC) has long been proposed to form the contours of place firing fields. Recent experiments largely, but not completely, support this theory. Place fields are obviously expanded by inactivation of dorsal MEC, which fits the hypothesis. However, contrary to the prediction, inactivation of ventral MEC is also weakly broaden the spatial place firing patterns. In this study, we derive the model from grid spatial frequencies represented by Gaussian profiles to a 1D place field by Bayesian inference, and further provide completely theoretical explanations for expansion of place fields and predictions for alignments of grid components. To understand the information transform across between neocortex, entorhinal cortex, and hippocampus, we propose spatial memory indexing theory from hippocampal indexing theory to investigate how neural dynamics work in the entorhinal-hippocampal circuit. The inputs of place cells in CA3 are converged from three grid modules with different grid spacings layer II of MEC by Bayesian mechanism. We resort to the robot system to test Fourier hypothesis and spatial memory indexing theory, and validate our proposed entorhinal-hippocampal model. And then we demonstrate its cognitive mapping capability on the KITTI odometry benchmark dataset. Results suggest that our model provides a rational theoretical explanation for the biological experimental results. Results also show that the proposed model is robust for simultaneous localization and mapping (SLAM) in the large-scale environment. Our proposed model theoretically supports for Fourier hypothesis in a general Bayesian mechanism, which may pertain to other neural systems in addition to spatial cognition.
Conventional financial models fail to explain the economic and monetary properties of cryptocurrencies due to the latter's dual nature: their usage as financial assets on the one side and their tight connection to the underlying blockchain structure on the other. In an effort to examine both components via a unified approach, we apply a recently developed Non-Homogeneous Hidden Markov (NHHM) model with an extended set of financial and blockchain specific covariates on the Bitcoin (BTC) and Ether (ETH) price data. Based on the observable series, the NHHM model offers a novel perspective on the underlying microstructure of the cryptocurrency market and provides insight on unobservable parameters such as the behavior of investors, traders and miners. The algorithm identifies two alternating periods (hidden states) of inherently different activity -- fundamental versus uninformed or noise traders -- in the Bitcoin ecosystem and unveils differences in both the short/long run dynamics and in the financial characteristics of the two states, such as significant explanatory variables, extreme events and varying series autocorrelation. In a somewhat unexpected result, the Bitcoin and Ether markets are found to be influenced by markedly distinct indicators despite their perceived correlation. The current approach backs earlier findings that cryptocurrencies are unlike any conventional financial asset and makes a first step towards understanding cryptocurrency markets via a more comprehensive lens.