Neural classifiers are non linear systems providing decisions on the classes of patterns, for a given problem they have learned. The output computed by a classifier for each pattern constitutes an approximation of the output of some unknown function, mapping pattern data to their respective classes. The lack of knowledge of such a function along with the complexity of neural classifiers, especially when these are deep learning architectures, do not permit to obtain information on how specific predictions have been made. Hence, these powerful learning systems are considered as black boxes and in critical applications their use tends to be considered inappropriate. Gaining insight on such a black box operation constitutes a one way approach in interpreting operation of neural classifiers and assessing the validity of their decisions. In this paper we tackle this problem introducing a novel methodology for discovering which features are considered relevant by a trained neural classifier and how they affect the classifier's output, thus obtaining an explanation on its decision. Although, feature relevance has received much attention in the machine learning literature here we reconsider it in terms of nonlinear parameter estimation targeted by a set membership approach which is based on interval analysis. Hence, the proposed methodology builds on sound mathematical approaches and the results obtained constitute a reliable estimation of the classifier's decision premises.
Although the recursive logit (RL) model has been recently popular and has led to many applications and extensions, an important numerical issue with respect to the computation of value functions remains unsolved. This issue is particularly significant for model estimation, during which the parameters are updated every iteration and may violate the feasibility condition of the value function. To solve this numerical issue of the value function in the model estimation, this study performs an extensive analysis of a prism-constrained RL (Prism-RL) model proposed by Oyama and Hato (2019), which has a path set constrained by the prism defined based upon a state-extended network representation. The numerical experiments have shown two important properties of the Prism-RL model for parameter estimation. First, the prism-based approach enables estimation regardless of the initial and true parameter values, even in cases where the original RL model cannot be estimated due to the numerical problem. We also successfully captured a positive effect of the presence of street green on pedestrian route choice in a real application. Second, the Prism-RL model achieved better fit and prediction performance than the RL model, by implicitly restricting paths with large detour or many loops. Defining the prism-based path set in a data-oriented manner, we demonstrated the possibility of the Prism-RL model describing more realistic route choice behavior. The capture of positive network attributes while retaining the diversity of path alternatives is important in many applications such as pedestrian route choice and sequential destination choice behavior, and thus the prism-based approach significantly extends the practical applicability of the RL model.
An approximate mean square error (MSE) expression for the performance analysis of implicitly defined estimators of non-random parameters is proposed. An implicitly defined estimator (IDE) declares the minimizer/maximizer of a selected cost/reward function as the parameter estimate. The maximum likelihood (ML) and the least squares estimators are among the well known examples of this class. In this paper, an exact MSE expression for implicitly defined estimators with a symmetric and unimodal objective function is given. It is shown that the expression reduces to the Cramer-Rao lower bound (CRLB) and misspecified CRLB in the large sample size regime for ML and misspecified ML estimation, respectively. The expression is shown to yield the Ziv-Zakai bound (without the valley filling function) for the maximum a posteriori (MAP) estimator when it is used in a Bayesian setting, that is, when an a-priori distribution is assigned to the unknown parameter. In addition, extension of the suggested expression to the case of nuisance parameters is studied and some approximations are given to ease the computations for this case. Numerical results indicate that the suggested MSE expression not only predicts the estimator performance in the asymptotic region; but it is also applicable for the threshold region analysis, even for IDEs whose objective functions do not satisfy the symmetry and unimodality assumptions. Advantages of the suggested MSE expression are its conceptual simplicity and its relatively straightforward numerical calculation due to the reduction of the estimation problem to a binary hypothesis testing problem, similar to the usage of Ziv-Zakai bounds in random parameter estimation problems.
One of the most important problems in system identification and statistics is how to estimate the unknown parameters of a given model. Optimization methods and specialized procedures, such as Empirical Minimization (EM) can be used in case the likelihood function can be computed. For situations where one can only simulate from a parametric model, but the likelihood is difficult or impossible to evaluate, a technique known as the Two-Stage (TS) Approach can be applied to obtain reliable parametric estimates. Unfortunately, there is currently a lack of theoretical justification for TS. In this paper, we propose a statistical decision-theoretical derivation of TS, which leads to Bayesian and Minimax estimators. We also show how to apply the TS approach on models for independent and identically distributed samples, by computing quantiles of the data as a first step, and using a linear function as the second stage. The proposed method is illustrated via numerical simulations.
Model-based optimization approaches for monitoring and control, such as model predictive control and optimal state and parameter estimation, have been used for decades in many engineering applications. Models describing the dynamics, constraints, and desired performance criteria are fundamental to model-based approaches. Thanks to recent technological advancements in digitalization, machine learning methods such as deep learning, and computing power, there has been an increasing interest in using machine learning methods alongside model-based approaches for control and estimation. The number of new methods and theoretical findings using machine learning for model-based control and optimization is increasing rapidly. This paper outlines the basic ideas and principles behind an easy-to-use Python toolbox that allows to quickly and efficiently solve machine-learning-supported optimization, model predictive control, and estimation problems. The toolbox leverages state-of-the-art machine learning libraries to train components used to define the problem. It allows to efficiently solve the resulting optimization problems. Machine learning can be used for a broad spectrum of tasks, ranging from model predictive control for stabilization, setpoint tracking, path following, and trajectory tracking to moving horizon estimation and Kalman filtering. For linear systems, it enables quick code generation for embedded MPC applications. HILO-MPC is flexible and adaptable, making it especially suitable for research and fundamental development tasks. Due to its simplicity and numerous already implemented examples, it is also a powerful teaching tool. The usability is underlined, presenting a series of application examples.
As several previous works have pointed out, the evolution of the wireless channels in multiple input multiple output systems can be advantageously modeled as an autoregressive process. Therefore, estimating the coefficients, and, in particular, the state transition matrix of this autoregressive process is a key to accurate channel estimation, tracking, and prediction in fast fading environments. In this paper we assume a time varying spatially uncorrelated channel, which is approximately the case with proper antenna spacing at the base station in rich scattering environments. We propose a method for autoregressive parameter estimation for the single input multiple output (SIMO) channel. We show an almost sure convergence of the estimated coefficients to the true autoregressive coefficients in large dimensions. We apply the proposed method to SIMO channel tracking.
In this note, we develop an adaptive observer for a class of nonlinear systems with switched unknown parameters to estimate the states and parameters simultaneously. The main challenge lies in how to eliminate the disturbance effect of zero-input responses caused by the switching on the parameter estimation. These responses depend on the unknown states at switching instants (SASI) and constitute an additive disturbance to the parameter estimation, which obstructs parameter convergence to zero. Our solution is to treat the zero-input responses as excitations instead of disturbances. This is realized by first augmenting the system parameter with the SASI and then developing an estimator for the augmented parameter using the \textit{dynamic regression extension and mixing} (DREM) technique. Thanks to its property of element-wise parameter adaptation, the system parameter estimation is decoupled from the SASI. As a result, the estimation errors of system states and parameters converge to zero asymptotically. Furthermore, the robustness of the proposed adaptive observer is guaranteed in the presence of disturbances and noise. A numerical example validates the effectiveness of the proposed approach.
Limit cycle oscillations are phenomena arising in nonlinear dynamical systems and characterized by periodic, locally-stable, and self-sustained state trajectories. Systems controlled in a closed loop along a periodic trajectory can also be modelled as systems experiencing limit cycle behavior. The goal of this work is to identify from data, the local dynamics around the limit cycle using linear periodically parameter-varying models. Using a coordinate transformation onto transversal surfaces, the dynamics are decomposed into two parts: one along the limit cycle, and one on the transversal surfaces. Then, the model is identified from trajectory data using kernel-based methods with a periodic kernel design. The kernel-based model is extended to also account for variations in system parameters associated with different operating conditions. The performance of the proposed identification method is demonstrated on a benchmark nonlinear system and on a simplified airborne wind energy model. The method provides accurate model parameter estimation, compared to the analytical linearization, and good prediction capability.
Dynamic state and parameter estimation (DSE) plays a key role for reliably monitoring and operating future, power-electronics-dominated power systems. While DSE is a very active research field, experimental applications of proposed algorithms to real-world systems remain scarce. This motivates the present paper, in which we demonstrate the effectiveness of a DSE algorithm previously presented by parts of the authors with real-world data collected by a Phasor Measurement Unit (PMU) at a substation close to a power plant within the extra-high voltage grid of Germany. To this end, at first we derive a suitable mapping of the real-world PMU-measurements recorded at a substation close to the power plant to the terminal bus of the power plants' synchronous generator (SG). This mapping considers the high-voltage (HV) transmission line, the tap-changing transformer and the auxiliary system of the power plant. Next, we introduce several practically motivated extensions to the estimation algorithm, which significantly improve its practical performance with real-world measurements. Finally, we successfully validate the algorithm experimentally in an auto- as well as a cross-validation.
This paper is about computationally tractable methods for power system parameter estimation and Optimal Experiment Design (OED). Here, the main motivation is that OED has the potential to significantly increase the accuracy of power system parameter estimates, for example, if only a few batches of data are available. The problem is, however, that solving the exact OED problem for larger power grids turns out to be computationally expensive and, in many cases, even computationally intractable. Therefore, the present paper proposes three numerical approximation techniques, which increase the computational tractability of OED for power systems. These approximation techniques are bench-marked on a 5-bus and a 14-bus case studies.
This paper addresses the problem of real-time monitoring of long-term voltage instability (LTVI) by using local field measurements. Existing local measurement-based methods use Thevenin equivalent parameter estimation that is sensitive to the noise in measurements. For solving this issue, we avoid the Thevenin approach by projecting the power flow equations as circles to develop local static-voltage stability indicator (LS-VSI) that is robust to measurement noises. The proposed method is an attractive option for practical implementation because of its decentralized nature, robustness to noise, and realistic modeling. Next, we utilize LS-VSI to propose a new local dynamic - VSI (LD-VSI) to identify the LTVI triggered by large disturbances and load tap changer dynamics. LD-VSI can identify not only the onset of instability with an alarm but also the voltage collapse point (VCP). Extensive numerical validation comparing LS-VSI with existing methods is presented on IEEE 30-bus and larger test systems like 2000-bus Texas synthetic grid to validate the robustness, accuracy, and situational awareness feature. We also verify the LD-VSI behavior on the Nordic power grid using PSSE dynamic simulation. When compared to existing methods, we observe that LD-VSI is not only more robust to measurement noise but also can identify VCP.
Adequate therapeutic retinal laser irradiation needs to be adapted to the local absorption. This leads to time-consuming treatments as the laser power needs to be successively adjusted to avoid under- and overtreatment caused by too low or too high temperatures. Closed-loop control can overcome this burden by means of temperature measurements. To allow for model predictive control schemes, the current state and the spot-dependent absorption need to be estimated. In this paper, we thoroughly compare moving horizon estimator (MHE) and extended Kalman filter (EKF) designs for joint state and parameter estimation. We consider two different scenarios, the estimation of one or two unknown absorption coefficients. For one unknown parameter, both estimators perform very similar. For two unknown parameters, we found that the MHE benefits from active parameter constraints at the beginning of the estimation, whereas after a settling time both estimators perform again very similar as long as the parameters are inside the considered parameter bounds.
In this paper, we analyze the generalization performance of the Iterative Hard Thresholding (IHT) algorithm widely used for sparse recovery problems. The parameter estimation and sparsity recovery consistency of IHT has long been known in compressed sensing. From the perspective of statistical learning, another fundamental question is how well the IHT estimation would predict on unseen data. This paper makes progress towards answering this open question by introducing a novel sparse generalization theory for IHT under the notion of algorithmic stability. Our theory reveals that: 1) under natural conditions on the empirical risk function over $n$ samples of dimension $p$, IHT with sparsity level $k$ enjoys an $\mathcal{\tilde O}(n^{-1/2}\sqrt{k\log(n)\log(p)})$ rate of convergence in sparse excess risk; 2) a tighter $\mathcal{\tilde O}(n^{-1/2}\sqrt{\log(n)})$ bound can be established by imposing an additional iteration stability condition on a hypothetical IHT procedure invoked to the population risk; and 3) a fast rate of order $\mathcal{\tilde O}\left(n^{-1}k(\log^3(n)+\log(p))\right)$ can be derived for strongly convex risk function under proper strong-signal conditions. The results have been substantialized to sparse linear regression and sparse logistic regression models to demonstrate the applicability of our theory. Preliminary numerical evidence is provided to confirm our theoretical predictions.
Various real-world scientific applications involve the mathematical modeling of complex uncertain systems with numerous unknown parameters. Accurate parameter estimation is often practically infeasible in such systems, as the available training data may be insufficient and the cost of acquiring additional data may be high. In such cases, based on a Bayesian paradigm, we can design robust operators retaining the best overall performance across all possible models and design optimal experiments that can effectively reduce uncertainty to enhance the performance of such operators maximally. While objective-based uncertainty quantification (objective-UQ) based on MOCU (mean objective cost of uncertainty) provides an effective means for quantifying uncertainty in complex systems, the high computational cost of estimating MOCU has been a challenge in applying it to real-world scientific/engineering problems. In this work, we propose a novel scheme to reduce the computational cost for objective-UQ via MOCU based on a data-driven approach. We adopt a neural message-passing model for surrogate modeling, incorporating a novel axiomatic constraint loss that penalizes an increase in the estimated system uncertainty. As an illustrative example, we consider the optimal experimental design (OED) problem for uncertain Kuramoto models, where the goal is to predict the experiments that can most effectively enhance robust synchronization performance through uncertainty reduction. We show that our proposed approach can accelerate MOCU-based OED by four to five orders of magnitude, without any visible performance loss compared to the state-of-the-art. The proposed approach applies to general OED tasks, beyond the Kuramoto model.
When directly coupled with fluctuating energy sources such as wind and photovoltage power, the alkaline electrolysis (AEL) in a power-to-hydrogen (P2H) system is required to operate flexibly by dynamically adjusting its hydrogen production rate. The flex-ibility characteristics, e.g., loading range and ramping rate, of an AEL system are significantly influenced by some parameters re-lated to the dynamic processes of the AEL system. These parame-ters are usually difficult to measure directly and may even change with time. To accurately evaluate the flexibility of an AEL system in online operation, this paper presents a Bayesian Inference-based Markov Chain Monte Carlo (MCMC) method to estimate these parameters. Meanwhile, posterior joint probability distribu-tions of the estimated parameters are obtained as a byproduct, which provides valuable physical insight into the AEL systems. Experiments on a 25 kW electrolyzer validate the proposed pa-rameter estimation method.
A topic model is often formulated as a generative model that explains how each word of a document is generated given a set of topics and document-specific topic proportions. It is focused on capturing the word co-occurrences in a document and hence often suffers from poor performance in analyzing short documents. In addition, its parameter estimation often relies on approximate posterior inference that is either not scalable or suffers from large approximation error. This paper introduces a new topic-modeling framework where each document is viewed as a set of word embedding vectors and each topic is modeled as an embedding vector in the same embedding space. Embedding the words and topics in the same vector space, we define a method to measure the semantic difference between the embedding vectors of the words of a document and these of the topics, and optimize the topic embeddings to minimize the expected difference over all documents. Experiments on text analysis demonstrate that the proposed method, which is amenable to mini-batch stochastic gradient descent based optimization and hence scalable to big corpora, provides competitive performance in discovering more coherent and diverse topics and extracting better document representations.
Biophysical modelling of the diffusion MRI signal provides estimates of specific microstructural tissue properties. Although nonlinear optimization such as non-linear least squares (NLLS) is the most widespread method for model estimation, it suffers from local minima and high computational cost. Deep Learning approaches are steadily replacing NL fitting, but come with the limitation that the model needs to be retrained for each acquisition protocol and noise level. The White Matter Tract Integrity (WMTI)-Watson model was proposed as an implementation of the Standard Model of diffusion in white matter that estimates model parameters from the diffusion and kurtosis tensors (DKI). Here we proposed a deep learning approach based on the encoder-decoder recurrent neural network (RNN) to increase the robustness and accelerate the parameter estimation of WMTI-Watson. We use an embedding approach to render the model insensitive to potential differences in distributions between training data and experimental data. This RNN-based solver thus has the advantage of being highly efficient in computation and more readily translatable to other datasets, irrespective of acquisition protocol and underlying parameter distributions as long as DKI was pre-computed from the data. In this study, we evaluated the performance of NLLS, the RNN-based method and a multilayer perceptron (MLP) on synthetic and in vivo datasets of rat and human brain. We showed that the proposed RNN-based fitting approach had the advantage of highly reduced computation time over NLLS (from hours to seconds), with similar accuracy and precision but improved robustness, and superior translatability to new datasets over MLP.
Laser photocoagulation is one of the most frequently used treatment approaches for retinal diseases such as diabetic retinopathy and macular edema. The use of model-based control, such as Model Predictive Control (MPC), enhances a safe and effective treatment by guaranteeing temperature bounds. In general, real-time requirements for model-based control designs are not met since the temperature distribution in the eye fundus is governed by a heat equation with a nonlinear parameter dependency. This issue is circumvented by representing the model by a lower-dimensional system which well-approximates the original model, including the parametric dependency. We combine a global-basis approach with the discrete empirical interpolation method, tailor its hyperparameters to laser photocoagulation, and show its superiority in comparison to a recently proposed method based on Taylor-series approximation. Its effectiveness is measured in computation time for MPC. We further present a case study to estimate the range of absorption parameters in porcine eyes, and by means of a theoretical and numerical sensitivity analysis we show that the sensitivity of the temperature increase is higher with respect to the absorption coefficient of the retinal pigment epithelium (RPE) than of the choroid's.
Pulmonary hypertension is a cardiovascular disorder manifested by elevated arterial blood pressure together with vessel wall stiffening and thickening due to alterations in collagen, elastin and smooth muscle cells. Hypoxia-induced (type 3) pulmonary hypertension can be studied in animals exposed to a low oxygen environment for prolonged time periods leading to biomechanical alterations in vessel wall structure. This study formulates and systematically reduces a nonlinear elastic structural wall model for a large pulmonary artery, generating a novel pressure-area relation capturing remodeling in type 3 pulmonary hypertension. The model is calibrated using {\em ex vivo} measurements of vessel diameter and wall thickness changes, under controlled flow conditions, in left pulmonary arteries isolated from control and hypertensive mice. A two-layer, hyperelastic, anisotropic model incorporating residual stresses is formulated using the Holzapfel-Gasser-Ogden model. Complex relations predicting vessel area and wall thickness with increasing blood pressure are derived and calibrated using the data. Sensitivity analysis, parameter estimation and subset selection are used to systematically reduce the 16-parameter model to one in which a much smaller subset of identifiable parameters is estimated via solution of an inverse problem. Our final reduced model includes a single set of three elastic moduli. Estimated ranges of these parameters demonstrate that nonlinear stiffening is dominated by elastin in the control animals and by collagen in the hypertensive group. The novel pressure-area relation developed in this study has potential impact on one-dimensional fluids network models of vessel wall remodeling in the presence of cardiovascular disease.
Adverse posttraumatic neuropsychiatric sequelae (APNS) are common among veterans and millions of Americans after traumatic exposures, resulting in substantial burdens for trauma survivors and society. Despite numerous studies conducted on APNS over the past decades, there has been limited progress in understanding the underlying neurobiological mechanisms due to several unique challenges. One of these challenges is the reliance on subjective self-report measures to assess APNS, which can easily result in measurement errors and biases (e.g., recall bias). To mitigate this issue, in this paper, we investigate the potential of leveraging the objective longitudinal mobile device data to identify homogeneous APNS states and study the dynamic transitions and potential risk factors of APNS after trauma exposure. To handle specific challenges posed by longitudinal mobile device data, we developed exploratory hidden Markov factor models and designed a Stabilized Expectation-Maximization algorithm for parameter estimation. Simulation studies were conducted to evaluate the performance of parameter estimation and model selection. Finally, to demonstrate the practical utility of the method, we applied it to mobile device data collected from the Advancing Understanding of RecOvery afteR traumA (AURORA) study.