We present a comprehensive framework for fusing measurements from multiple and generally placed accelerometers and gyroscopes to perform inertial navigation. Using the angular acceleration provided by the accelerometer array, we show that the numerical integration of the orientation can be done with second-order accuracy, which is more accurate compared to the traditional first-order accuracy that can be achieved when only using the gyroscopes. Since orientation errors are the most significant error source in inertial navigation, improving the orientation estimation reduces the overall navigation error. The practical performance benefit depends on prior knowledge of the inertial sensor array, and therefore we present four different state-space models using different underlying assumptions regarding the orientation modeling. The models are evaluated using a Lie Group Extended Kalman filter through simulations and real-world experiments. We also show how individual accelerometer biases are unobservable and can be replaced by a six-dimensional bias term whose dimension is fixed and independent of the number of accelerometers.
The paper provides a novel framework to study the accuracy and stability of numerical integration schemes when employed for the time domain simulation of power systems. A matrix pencil-based approach is adopted to evaluate the error between the dynamic modes of the power system and the modes of the approximated discrete-time system arising from the application of the numerical method. The proposed approach can provide meaningful insights on how different methods compare to each other when applied to a power system, while being general enough to be systematically utilized for, in principle, any numerical method. The framework is illustrated for a handful of well-known explicit and implicit methods, while simulation results are presented based on the WSCC 9-bus system, as well as on a 1, 479-bus dynamic model of the All-Island Irish Transmission System.
This paper develops a sequential convex programming approach for Mars entry trajectory planning by range discretization. To improve the accuracy of numerical integration, the range of entry trajectory is selected as the independent variable rather than time or energy. A dilation factor is employed to normalize the entry dynamics and integration interval of the performance index so that the difficult free-final-time programming problem can be converted to a fixed-final-range optimization problem. The bank angle rate with respect to the range is introduced as the new control input in order to decouple the control from the state and facilitate convexification of constraints on the bank angle and its rate. The nonlinear bank angle rate constraint is further relaxed into a linear one via inequality relaxation. Moreover, the nonconvex minimum-time performance index is convexified by regarding flight time as a state variable. Then, the Mars entry trajectory planning problem can be formulated into the framework of convex programming after linearization. By range discretization and successive convexification, the reformulated Mars entry trajectory planning problem is transcribed into a series of convex optimization sub-problems that can be sequentially solved using the convex programming algorithm. The virtual control and adaptive trust-region techniques are employed to improve the accuracy, robustness, and computation efficiency of the algorithm. Numerical simulations with comparative studies are presented to demonstrate the convergence performance and efficiency of the proposed algorithm.
Learned iterative reconstruction algorithms for inverse problems offer the flexibility to combine analytical knowledge about the problem with modules learned from data. This way, they achieve high reconstruction performance while ensuring consistency with the measured data. In computed tomography, extending such approaches from 2D fan-beam to 3D cone-beam data is challenging due to the prohibitively high GPU memory that would be needed to train such models. This paper proposes to use neural ordinary differential equations to solve the reconstruction problem in a residual formulation via numerical integration. For training, there is no need to backpropagate through several unrolled network blocks nor through the internals of the solver. Instead, the gradients are obtained very memory-efficiently in the neural ODE setting allowing for training on a single consumer graphics card. The method is able to reduce the root mean squared error by over 30% compared to the best performing classical iterative reconstruction algorithm and produces high quality cone-beam reconstructions even in a sparse view scenario.
Most recently, machine learning has been used to study the dynamics of integrable Hamiltonian systems and the chaotic 3-body problem. In this work, we consider an intermediate case of regular motion in a non-integrable system: the behaviour of objects in the 2:3 mean motion resonance with Neptune. We show that, given initial data from a short 6250 yr numerical integration, the best-trained artificial neural network (ANN) can predict the trajectories of the 2:3 resonators over the subsequent 18750 yr evolution, covering a full libration cycle over the combined time period. By comparing our ANN's prediction of the resonant angle to the outcome of numerical integrations, the former can predict the resonant angle with an accuracy as small as of a few degrees only, while it has the advantage of considerably saving computational time. More specifically, the trained ANN can effectively measure the resonant amplitudes of the 2:3 resonators, and thus provides a fast approach that can identify the resonant candidates. This may be helpful in classifying a huge population of KBOs to be discovered in future surveys.
Neural ordinary differential equations (NODEs) -- parametrizations of differential equations using neural networks -- have shown tremendous promise in learning models of unknown continuous-time dynamical systems from data. However, every forward evaluation of a NODE requires numerical integration of the neural network used to capture the system dynamics, making their training prohibitively expensive. Existing works rely on off-the-shelf adaptive step-size numerical integration schemes, which often require an excessive number of evaluations of the underlying dynamics network to obtain sufficient accuracy for training. By contrast, we accelerate the evaluation and the training of NODEs by proposing a data-driven approach to their numerical integration. The proposed Taylor-Lagrange NODEs (TL-NODEs) use a fixed-order Taylor expansion for numerical integration, while also learning to estimate the expansion's approximation error. As a result, the proposed approach achieves the same accuracy as adaptive step-size schemes while employing only low-order Taylor expansions, thus greatly reducing the computational cost necessary to integrate the NODE. A suite of numerical experiments, including modeling dynamical systems, image classification, and density estimation, demonstrate that TL-NODEs can be trained more than an order of magnitude faster than state-of-the-art approaches, without any loss in performance.
Next generation reservoir computing based on nonlinear vector autoregression (NVAR) is applied to emulate simple dynamical system models and compared to numerical integration schemes such as Euler and the $2^\text{nd}$ order Runge-Kutta. It is shown that the NVAR emulator can be interpreted as a data-driven method used to recover the numerical integration scheme that produced the data. It is also shown that the approach can be extended to produce high-order numerical schemes directly from data. The impacts of the presence of noise and temporal sparsity in the training set is further examined to gauge the potential use of this method for more realistic applications.
Bayesian quadrature (BQ) is a model-based numerical integration method that is able to increase sample efficiency by encoding and leveraging known structure of the integration task at hand. In this paper, we explore priors that encode invariance of the integrand under a set of bijective transformations in the input domain, in particular some unitary transformations, such as rotations, axis-flips, or point symmetries. We show initial results on superior performance in comparison to standard Bayesian quadrature on several synthetic and one real world application.
We propose a novel deterministic sampling method to approximate a target distribution $\rho^*$ by minimizing the kernel discrepancy, also known as the Maximum Mean Discrepancy (MMD). By employing the general \emph{energetic variational inference} framework (Wang et al., 2021), we convert the problem of minimizing MMD to solving a dynamic ODE system of the particles. We adopt the implicit Euler numerical scheme to solve the ODE systems. This leads to a proximal minimization problem in each iteration of updating the particles, which can be solved by optimization algorithms such as L-BFGS. The proposed method is named EVI-MMD. To overcome the long-existing issue of bandwidth selection of the Gaussian kernel, we propose a novel way to specify the bandwidth dynamically. Through comprehensive numerical studies, we have shown the proposed adaptive bandwidth significantly improves the EVI-MMD. We use the EVI-MMD algorithm to solve two types of sampling problems. In the first type, the target distribution is given by a fully specified density function. The second type is a "two-sample problem", where only training data are available. The EVI-MMD method is used as a generative learning model to generate new samples that follow the same distribution as the training data. With the recommended settings of the tuning parameters, we show that the proposed EVI-MMD method outperforms some existing methods for both types of problems.
When approximating the expectations of a functional of a solution to a stochastic differential equation, the numerical performance of deterministic quadrature methods, such as sparse grid quadrature and quasi-Monte Carlo (QMC) methods, may critically depend on the regularity of the integrand. To overcome this issue and improve the regularity structure of the problem, we consider cases in which analytic smoothing (bias-free mollification) cannot be performed and introduce a novel numerical smoothing approach by combining a root-finding method with a one-dimensional numerical integration with respect to a single well-chosen variable. We prove that, under appropriate conditions, the resulting function of the remaining variables is highly smooth, potentially affording the improved efficiency of adaptive sparse grid quadrature (ASGQ) and QMC methods, particularly when combined with hierarchical transformations (ie., the Brownian bridge and Richardson extrapolation on the weak error). This approach facilitates the effective treatment of high dimensionality. Our study is motivated by option pricing problems, focusing on dynamics where the discretization of the asset price is necessary. Based on our analysis and numerical experiments, we demonstrate the advantages of combining numerical smoothing with the ASGQ and QMC methods over these methods without smoothing and the Monte Carlo approach. Finally, our approach is generic and can be applied to solve a broad class of problems, particularly approximating distribution functions, computing financial Greeks, and estimating risk quantities.
This thesis proposes novel Small-Signal Stability Analysis (SSSA)-based techniques that contribute to electric power system modal analysis, automatic control, and numerical integration. Modal analysis is a fundamental tool for power system stability analysis and control. The thesis proposes a SSSA approach to determine the Participation Factors (PFs) of algebraic variables in power system dynamic modes. The thesis also explores SSSA techniques for the design of power system controllers. The contributions on this topic are twofold: i) Investigate a promising control approach, that is to synthesize automatic regulators for power systems based on the theory of fractional calculus. ii) Propose a novel perspective on the potential impact of time delays on power system stability. Through SSSA, the thesis systematically identifies the control parameter settings for which delays in PSSs improve the damping of a power system. Both analytical and simulation-based results are presented. Finally, SSSA is utilized in the thesis to systematically propose a delay-based method to reduce the coupling of the equations of power system models for transient stability analysis. The method consists in identifying the variables that, when subjected to a delay equal to the time step of the numerical integration, leave practically unchanged the system trajectories. Such an one-step-delay approximation increases the sparsity of the system Jacobian matrices and can be used in conjunction with state-of-the-art techniques for the integration of DAEs. The proposed approach is evaluated in terms of accuracy, convergence and computational burden. Throughout the thesis, the proposed techniques are duly validated through numerical tests based on real-world network models.
The dynamics of power grids are governed by a large number of nonlinear differential and algebraic equations (DAEs). To safely operate the system, operators need to check that the states described by these DAEs stay within prescribed limits after various potential faults. However, current numerical solvers of DAEs are often too slow for real-time system operations. In addition, detailed system parameters are often not exactly known. Machine learning approaches have been proposed to reduce computational efforts, but existing methods generally suffer from overfitting and failures to predict unstable behaviors. This paper proposes a novel framework to predict power system transients by learning in the frequency domain. The intuition is that although the system behavior is complex in the time domain, there are relatively few dominant modes in the frequency domain. Therefore, we learn to predict by constructing neural networks with Fourier transform and filtering layers. System topology and fault information are encoded by taking a multi-dimensional Fourier transform, allowing us to leverage the fact that the trajectories are sparse both in time and spatial frequencies. We show that the proposed approach does not need detailed system parameters, greatly speeds up prediction computations and is highly accurate for different fault types.
In the context of high penetration of renewables, the need to build dynamic models of power system components based on accessible measurement data has become urgent. To address this challenge, firstly, a neural ordinary differential equations (ODE) module and a neural differential-algebraic equations (DAE) module are proposed to form a data-driven modeling framework that accurately captures components' dynamic characteristics and flexibly adapts to various interface settings. Secondly, analytical models and data-driven models learned by the neural ODE and DAE modules are integrated together and simulated simultaneously using unified transient stability simulation methods. Finally, the neural ODE and DAE modules are implemented with Python and made public on GitHub. Using the portal measurements, three simple but representative cases of excitation controller modeling, photovoltaic power plant modeling, and equivalent load modeling of a regional power network are carried out in the IEEE-39 system and 2383wp system. Neural dynamic model-integrated simulations are compared with the original model-based ones to verify the feasibility and potentiality of the proposed neural ODE and DAE modules.
A least-squares neural network (LSNN) method was introduced for solving scalar linear and nonlinear hyperbolic conservation laws (HCLs) in [7, 6]. This method is based on an equivalent least-squares (LS) formulation and uses ReLU neural network as approximating functions, making it ideal for approximating discontinuous functions with unknown interface location. In the design of the LSNN method for HCLs, the numerical approximation of differential operators is a critical factor, and standard numerical or automatic differentiation along coordinate directions can often lead to a failed NN-based method. To overcome this challenge, this paper rewrites HCLs in their divergence form of space and time and introduces a new discrete divergence operator. As a result, the proposed LSNN method is free of penalization of artificial viscosity. Theoretically, the accuracy of the discrete divergence operator is estimated even for discontinuous solutions. Numerically, the LSNN method with the new discrete divergence operator was tested for several benchmark problems with both convex and non-convex fluxes, and was able to compute the correct physical solution for problems with rarefaction, shock or compound waves. The method is capable of capturing the shock of the underlying problem without oscillation or smearing, even without any penalization of the entropy condition, total variation, and/or artificial viscosity.
The problem of determining the underlying dynamics of a system when only given data of its state over time has challenged scientists for decades. In this paper, the approach of using machine learning to model the updates of the phase space variables is introduced; this is done as a function of the phase space variables. (More generally, the modeling is done over functions of the jet space.) This approach (named FJet) allows one to accurately replicate the dynamics, and is demonstrated on the examples of the damped harmonic oscillator, the damped pendulum, and the Duffing oscillator; the underlying differential equation is also accurately recovered for each example. In addition, the results in no way depend on how the data is sampled over time (i.e., regularly or irregularly). It is demonstrated that a regression implementation of FJet is similar to the model resulting from a Taylor series expansion of the Runge-Kutta (RK) numerical integration scheme. This identification confers the advantage of explicitly revealing the function space to use in the modeling, as well as the associated uncertainty quantification for the updates. Finally, it is shown in the undamped harmonic oscillator example that the stability of the updates is stable $10^9$ times longer than with $4$th-order RK (with time step $0.1$).
Fluidic locomotion of flapping Micro Aerial Vehicles (MAVs) can be very complex, particularly when the rules from insect flight dynamics (fast flapping dynamics and light wings) are not applicable. In these situations, widely used averaging techniques can fail quickly. The primary motivation is to find efficient models for complex forms of aerial locomotion where wings constitute a large part of body mass (i.e., dominant inertial effects) and deform in multiple directions (i.e., morphing wing). In these systems, high degrees of freedom yields complex inertial, Coriolis, and gravity terms. We use Algorithmic Differentiation (AD) and Bayesian filters computed with cubature rules conjointly to quickly estimate complex fluid-structure interactions. In general, Bayesian filters involve finding complex numerical integration (e.g., find posterior integrals). Using cubature rules to compute Gaussian-weighted integrals and AD, we show that the complex multi-degrees-of-freedom dynamics of morphing MAVs can be computed very efficiently and accurately. Therefore, our work facilitates closed-loop feedback control of these morphing MAVs.
Bayesian inference in biological modeling commonly relies on Markov chain Monte Carlo (MCMC) sampling of a multidimensional and non-Gaussian posterior distribution that is not analytically tractable. Here, we present the implementation of a practical MCMC method in the open-source software package PyBioNetFit (PyBNF), which is designed to support parameterization of mathematical models for biological systems. The new MCMC method, am, incorporates an adaptive move proposal distribution. For warm starts, sampling can be initiated at a specified location in parameter space and with a multivariate Gaussian proposal distribution defined initially by a specified covariance matrix. Multiple chains can be generated in parallel using a computer cluster. We demonstrate that am can be used to successfully solve real-world Bayesian inference problems, including forecasting of new Coronavirus Disease 2019 case detection with Bayesian quantification of forecast uncertainty. PyBNF version 1.1.9, the first stable release with am, is available at PyPI and can be installed using the pip package-management system on platforms that have a working installation of Python 3. PyBNF relies on libRoadRunner and BioNetGen for simulations (e.g., numerical integration of ordinary differential equations defined in SBML or BNGL files) and Dask.Distributed for task scheduling on Linux computer clusters.
Measurement noise is an integral part while collecting data of a physical process. Thus, noise removal is a necessary step to draw conclusions from these data, and it often becomes quite essential to construct dynamical models using these data. We discuss a methodology to learn differential equation(s) using noisy and sparsely sampled measurements. In our methodology, the main innovation can be seen in of integration of deep neural networks with a classical numerical integration method. Precisely, we aim at learning a neural network that implicitly represents the data and an additional neural network that models the vector fields of the dependent variables. We combine these two networks by enforcing the constraint that the data at the next time-steps can be given by following a numerical integration scheme such as the fourth-order Runge-Kutta scheme. The proposed framework to learn a model predicting the vector field is highly effective under noisy measurements. The approach can handle scenarios where dependent variables are not available at the same temporal grid. We demonstrate the effectiveness of the proposed method to learning models using data obtained from various differential equations. The proposed approach provides a promising methodology to learn dynamic models, where the first-principle understanding remains opaque.
Cells sense their surrounding by employing intracellular signaling pathways that transmit hormonal signals from the cell membrane to the nucleus. TGF-$\beta$/SMAD signaling encodes various cell fates, controls tissue homeostasis and is deregulated in diseases such as cancer. The pathway shows strong heterogeneity at the single-cell level, but quantitative insights into mechanisms underlying fluctuations at various time scales are still missing, partly due to inefficiency in the calibration of stochastic models that mechanistically describe signaling processes. In this work we analyze single-cell TGF-$\beta$/SMAD signaling and show that it exhibits temporal stochastic bursts which are dose-dependent and whose number and magnitude correlate with cell migration. We propose a stochastic modeling approach to mechanistically describe these pathway fluctuations with high computational efficiency. Employing high-order numerical integration and fitting to burst statistics we enable efficient quantitative parameter estimation and discriminate models that assume noise in different reactions at the receptor level. This modeling approach suggests that stochasticity in the internalization of TGF-$\beta$ receptors into endosomes plays a key role in the observed temporal bursting. Further, the model predicts the single-cell dynamics of TGF-$\beta$/SMAD signaling in untested conditions, e.g., successfully reflects memory effects of signaling noise and cellular sensitivity towards repeated stimulation. Taken together, our computational framework based on burst analysis, noise modeling and path computation scheme is a suitable tool for the data-based modeling of complex signaling pathways, capable of identifying the source of temporal noise.
Computation of general state- and/or control-constrained Optimal Control Problems (OCPs) is difficult for various constraints, especially the intractable path constraint. For such problems, the theoretical convergence of numerical algorithms is usually not guaranteed, and the right solution may not be successfully obtained. With the recently proposed Variation Evolving Method (VEM), the evolution equations, which guarantee the convergence towards the optimal solution in theory even for the general constrained OCPs, are derived. In particular, the costate-free optimality conditions are established. Besides the analytic expressions of the costates and the Lagrange multipliers adjoining the terminal constraint, the integral equation that determines the Karush-Kuhn-Tucker (KKT) multiplier variable is also derived. Upon the work in this paper, the general constrained OCPs may be transformed to the Initial-value Problems (IVPs) to be solved, with common Ordinary Differential Equation (ODE) numerical integration methods.