This paper considers the problem of computing Bayesian estimates of both states and model parameters for nonlinear state-space models. Generally, this problem does not have a tractable solution and approximations must be utilised. In this work, a variational approach is used to provide an assumed density which approximates the desired, intractable, distribution. The approach is deterministic and results in an optimisation problem of a standard form. Due to the parametrisation of the assumed density selected first- and second-order derivatives are readily available which allows for efficient solutions. The proposed method is compared against state-of-the-art Hamiltonian Monte Carlo in two numerical examples.
Let $V$ be any vector space of multivariate degree-$d$ homogeneous polynomials with co-dimension at most $k$, and $S$ be the set of points where all polynomials in $V$ {\em nearly} vanish. We establish a qualitatively optimal upper bound on the size of $\epsilon$-covers for $S$, in the $\ell_2$-norm. Roughly speaking, we show that there exists an $\epsilon$-cover for $S$ of cardinality $M = (k/\epsilon)^{O_d(k^{1/d})}$. Our result is constructive yielding an algorithm to compute such an $\epsilon$-cover that runs in time $\mathrm{poly}(M)$. Building on our structural result, we obtain significantly improved learning algorithms for several fundamental high-dimensional probabilistic models with hidden variables. These include density and parameter estimation for $k$-mixtures of spherical Gaussians (with known common covariance), PAC learning one-hidden-layer ReLU networks with $k$ hidden units (under the Gaussian distribution), density and parameter estimation for $k$-mixtures of linear regressions (with Gaussian covariates), and parameter estimation for $k$-mixtures of hyperplanes. Our algorithms run in time {\em quasi-polynomial} in the parameter $k$. Previous algorithms for these problems had running times exponential in $k^{\Omega(1)}$. At a high-level our algorithms for all these learning problems work as follows: By computing the low-degree moments of the hidden parameters, we are able to find a vector space of polynomials that nearly vanish on the unknown parameters. Our structural result allows us to compute a quasi-polynomial sized cover for the set of hidden parameters, which we exploit in our learning algorithms.
Machine learning algorithms have been successfully used to approximate nonlinear maps under weak assumptions on the structure and properties of the maps. We present deep neural networks using dense and convolutional layers to solve an inverse problem, where we seek to estimate parameters of a FitzHugh-Nagumo model, which consists of a nonlinear system of ordinary differential equations (ODEs). We employ the neural networks to approximate reconstruction maps for model parameter estimation from observational data, where the data comes from the solution of the ODE and takes the form of a time series representing dynamically spiking membrane potential of a biological neuron. We target this dynamical model because of the computational challenges it poses in an inference setting, namely, having a highly nonlinear and nonconvex data misfit term and permitting only weakly informative priors on parameters. These challenges cause traditional optimization to fail and alternative algorithms to exhibit large computational costs. We quantify the prediction errors of model parameters obtained from the neural networks and investigate the effects of network architectures with and without the presence of noise in observational data. We generalize our framework for neural network-based reconstruction maps to simultaneously estimate ODE parameters and parameters of autocorrelated observational noise. Our results demonstrate that deep neural networks have the potential to estimate parameters in dynamical models and stochastic processes, and they are capable of predicting parameters accurately for the FitzHugh-Nagumo model.
This paper considers parameter estimation for nonlinear state-space models, which is an important but challenging problem. We address this challenge by employing a variational inference (VI) approach, which is a principled method that has deep connections to maximum likelihood estimation. This VI approach ultimately provides estimates of the model as solutions to an optimisation problem, which is deterministic, tractable and can be solved using standard optimisation tools. A specialisation of this approach for systems with additive Gaussian noise is also detailed. The proposed method is examined numerically on a range of simulated and real examples focusing on the robustness to parameter initialisation; additionally, favourable comparisons are performed against state-of-the-art alternatives.
Tracking is one of the most time consuming aspects of event reconstruction at the Large Hadron Collider (LHC) and its high-luminosity upgrade (HL-LHC). Innovative detector technologies extend tracking to four-dimensions by including timing in the pattern recognition and parameter estimation. However, present and future hardware already have additional information that is largely unused by existing track seeding algorithms. The shape of clusters provides an additional dimension for track seeding that can significantly reduce the combinatorial challenge of track finding. We use neural networks to show that cluster shapes can reduce significantly the rate of fake combinatorical backgrounds while preserving a high efficiency. We demonstrate this using the information in cluster singlets, doublets and triplets. Numerical results are presented with simulations from the TrackML challenge.
Despite the popularity of guitar effects, there is very little existing research on classification and parameter estimation of specific plugins or effect units from guitar recordings. In this paper, convolutional neural networks were used for classification and parameter estimation for 13 overdrive, distortion and fuzz guitar effects. A novel dataset of processed electric guitar samples was assembled, with four sub-datasets consisting of monophonic or polyphonic samples and discrete or continuous settings values, for a total of about 250 hours of processed samples. Results were compared for networks trained and tested on the same or on a different sub-dataset. We found that discrete datasets could lead to equally high performance as continuous ones, whilst being easier to design, analyse and modify. Classification accuracy was above 80\%, with confusion matrices reflecting similarities in the effects timbre and circuits design. With parameter values between 0.0 and 1.0, the mean absolute error is in most cases below 0.05, while the root mean square error is below 0.1 in all cases but one.
Growing demand for fast charging and optimised battery designs is fuelling significant interest in electrochemical models of Li-ion batteries. However, estimating parameter values for these models remains a major challenge. In this paper, a structural identifiability analysis was applied to a pseudo-2D Li-ion electrochemical battery model that can be considered as a linearised and decoupled form of the benchmark Doyle-Fuller-Newman model. From an inspection of the impedance function, it was shown that this model is uniquely parametrised by 21 parameters, being combinations of the electrochemical parameters like the conductivities and diffusion coefficients. The well-posedness of the parameter estimation problem with these parameters was then established. This result could lead to more realistic predictions about the internal state of the battery by identifying the parameter set that can be uniquely identified from the data.
Location information offered by external positioning systems, e.g., satellite navigation, can be used as prior information in the process of beam alignment and channel parameter estimation for reconfigurable intelligent surface (RIS)-aided millimeter wave (mmWave) multiple-input multiple-output networks. Benefiting from the availability of such prior information, albeit imperfect, the beam alignment and channel parameter estimation processes can be significantly accelerated with less candidate beams explored at all the terminals. We propose a practical channel parameter estimation method via atomic norm minimization, which outperforms the standard beam alignment in terms of both the mean square error and the effective spectrum efficiency for the same training overhead.
Motivated by future automotive applications, we study some joint radar target detection and parameter estimation problems where the transmitter, equipped with a mono-static MIMO radar, wishes to detect multiple targets and then estimate their respective parameters, while simultaneously communicating information data using orthogonal time frequency space (OTFS) modulation. Assuming that the number of radio frequency chains is smaller than the number of antennas over the mmWave frequency band, we design hybrid digital-analog beamforming at the radar transmitter adapted to different operating phases. The first scenario considers a wide angular beam in order to perform the target detection and parameter estimation, while multicasting a common message to all possible active users. The second scenario considers narrow angular beams to send information streams individually to the already detected users and simultaneously keep tracking of their respective parameters. Under this setup, we propose an efficient maximum likelihood scheme combined with hybrid beamforming to jointly perform target detection and parameter estimation. Our numerical results demonstrate that the proposed algorithm is able to reliably detect multiple targets with a sufficient number of antennas and achieves the Cram\'er-Rao lower bound for radar parameter estimation such as delay, Doppler and angle-of-arrival (AoA).
In this work, we develop and compare two innovative strategies for parameter estimation and radar detection of multiple point-like targets. The first strategy, which appears here for the first time, jointly exploits the maximum likelihood approach and Bayesian learning to estimate targets' parameters including their positions in terms of range bins. The second strategy relies on the intuition that for high signal-to-interference plus-noise ratio values, the energy of data containing target components projected onto the nominal steering direction should be higher than the energy of data affected by interference only. The adaptivity with respect to the interference covariance matrix is also considered exploiting a training data set collected in the proximity of the window under test. Finally, another important innovation aspect concerns the adaptive estimation of the unknown number of targets by means of the model order selection rules.
Quantization of signals is an integral part of modern signal processing applications, such as sensing, communication, and inference. While signal quantization provides many physical advantages, it usually degrades the subsequent estimation performance that is based on quantized data. In order to maintain physical constraints and simultaneously bring substantial performance gain, in this work we consider systems with mixed-resolution, 1-bit quantized and continuous-valued, data. First, we describe the linear minimum mean-squared error (LMMSE) estimator and its associated mean-squared error (MSE) for the general mixed-resolution model. However, the MSE of the LMMSE requires matrix inversion in which the number of measurements defines the matrix dimensions and thus, is not a tractable tool for optimization and system design. Therefore, we present the linear Gaussian orthonormal (LGO) measurement model and derive a closed-form analytic expression for the MSE of the LMMSE estimator under this model. In addition, we present two common special cases of the LGO model: 1) scalar parameter estimation and 2) channel estimation in mixed-ADC multiple-input multiple-output (MIMO) communication systems. We then solve the resource allocation optimization problem of the LGO model with the proposed tractable form of the MSE as an objective function and under a power constraint using a one-dimensional search. Moreover, we present the concept of dithering for mixed-resolution models and optimize the dithering noise as part of the resource allocation optimization problem for two dithering schemes: 1) adding noise only to the quantized measurements and 2) adding noise to both measurement types. Finally, we present simulations that demonstrate the advantages of using mixed-resolution measurements and the possible improvement introduced with dithering and resource allocation.
In this paper we address the problem of state observation of linear time-varying systems with delayed measurements, which has attracted the attention of many researchers|see [7] and references therein. We show that, adopting the parameter estimationbased approach proposed in [3,4], we can provide a very simple solution to the problem with reduced prior knowledge.
The multinomial probit model is a popular tool for analyzing choice behaviour as it allows for correlation between choice alternatives. Because current model specifications employ a full covariance matrix of the latent utilities for the choice alternatives, they are not scalable to a large number of choice alternatives. This paper proposes a factor structure on the covariance matrix, which makes the model scalable to large choice sets. The main challenge in estimating this structure is that the model parameters require identifying restrictions. We identify the parameters by a trace-restriction on the covariance matrix, which is imposed through a reparametrization of the factor structure. We specify interpretable prior distributions on the model parameters and develop an MCMC sampler for parameter estimation. The proposed approach significantly improves performance in large choice sets relative to existing multinomial probit specifications. Applications to purchase data show the economic importance of including a large number of choice alternatives in consumer choice analysis.
To achieve high data rates defined in 5G, the use of millimeter-waves and massive-MIMO are indispensable. To benefit from these technologies, an accurate estimation of the channel parameters is crucial. We propose a novel two-stage algorithm for channel parameters estimation. In the first stage, coarse estimation is accomplished by applying parameter estimation via interpolation based on DFT grid (PREIDG) with a fixed look-up table (LUT), while the second stage refines the estimates by means of the space-alternating generalized expectation maximization (SAGE) algorithm. The two-stage algorithm uses discrete Fourier transform beamforming vectors which are efficiently implemented by a Butler matrix in the analog domain. We found that this methodology improves the estimates compared to the auxiliary beam pair (ABP) method. The two-stage algorithm shows efficient performance in the low signal to noise ratio regime for the channel parameters i.e. angles of departure, complex path gains and delays of the multipaths. Finally, we derived the Cram\'er-Rao lower bound (CRLB) to assess the performance of our two-stage estimation algorithm.
We study finite-time horizon continuous-time linear-quadratic reinforcement learning problems in an episodic setting, where both the state and control coefficients are unknown to the controller. We first propose a least-squares algorithm based on continuous-time observations and controls, and establish a logarithmic regret bound of order $O((\ln M)(\ln\ln M))$, with $M$ being the number of learning episodes. The analysis consists of two parts: perturbation analysis, which exploits the regularity and robustness of the associated Riccati differential equation; and parameter estimation error, which relies on sub-exponential properties of continuous-time least-squares estimators. We further propose a practically implementable least-squares algorithm based on discrete-time observations and piecewise constant controls, which achieves similar logarithmic regret with an additional term depending explicitly on the time stepsizes used in the algorithm.
Mobile network is evolving from a communication-only network towards one with joint communication and radar/radio sensing (JCAS) capabilities, that we call perceptive mobile network (PMN). In PMNs, JCAS integrates sensing into communications, sharing a majority of system modules and the same transmitted signals. The PMN is expected to provide a ubiquitous radio sensing platform and enable a vast number of novel smart applications, whilst providing non-compromised communications. In this paper, we present a broad picture of the motivation, methodologies, challenges, and research opportunities of realizing PMN, by providing a comprehensive survey for systems and technologies developed mainly in the last ten years. Beginning by reviewing the work on coexisting communication and radar systems, we highlight their limits on addressing the interference problem, and then introduce the JCAS technology. We then set up JCAS in the mobile network context and envisage its potential applications. We continue to provide a brief review of three types of JCAS systems, with particular attention to their differences in design philosophy. We then introduce a framework of PMN, including the system platform and infrastructure, three types of sensing operations, and signals usable for sensing. Subsequently, we discuss required system modifications to enable sensing on current communication-only infrastructure. Within the context of PMN, we review stimulating research problems and potential solutions, organized under nine topics: performance bounds, waveform optimization, antenna array design, clutter suppression, sensing parameter estimation, resolution of sensing ambiguity, pattern analysis, networked sensing under cellular topology, and sensing-assisted communications. We conclude the paper by listing key open research problems for the aforementioned topics and sharing some lessons that we have learned.
A novel approach to solve the problem of distributed state estimation of linear time-invariant systems is proposed in this paper. It relies on the application of parameter estimation-based observers, where the state observation task is reformulated as a parameter estimation problem. In contrast with existing results our solution achieves convergence in finite-time, without injection of high gain, and imposes very weak assumptions on the communication graph---namely the existence of a Hamiltonian walk. The scheme is shown to be robust vis-\'a-vis external disturbances and communication delays.
We demonstrate the ability of statistical data assimilation to identify the measurements required for accurate state and parameter estimation in an epidemiological model for the novel coronavirus disease COVID-19. Our context is an effort to inform policy regarding social behavior, to mitigate strain on hospital capacity. The model unknowns are taken to be: the time-varying transmission rate, the fraction of exposed cases that require hospitalization, and the time-varying detection probabilities of new asymptomatic and symptomatic cases. In simulations, we obtain accurate estimates of undetected (that is, unmeasured) infectious populations, by measuring the detected cases together with the recovered and dead - and without assumed knowledge of the detection rates. Given a noiseless measurement of the recovered population, excellent estimates of all quantities are obtained using a temporal baseline of 101 days, with the exception of the time-varying transmission rate at times prior to the implementation of social distancing. With low noise added to the recovered population, accurate state estimates require a lengthening of the temporal baseline of measurements. Estimates of all parameters are sensitive to the contamination, highlighting the need for accurate and uniform methods of reporting. The aim of this paper is to exemplify the power of SDA to determine what properties of measurements will yield estimates of unknown parameters to a desired precision, in a model with the complexity required to capture important features of the COVID-19 pandemic.
The emergence of the Internet-of-Things and cyber-physical systems necessitates the coordination of access to limited communication resources in an autonomous and distributed fashion. Herein, the optimal design of a wireless sensing system with n sensors communicating with a fusion center via a collision channel of limited capacity k (k < n) is considered. In particular, it is shown that the problem of minimizing the mean-squared error subject to a threshold-based strategy at the transmitters is quasi-convex. As such, low complexity, numerical optimization methods can be applied. When coordination among sensors is not possible, the performance of the optimal threshold strategy is close to that of a centralized lower bound. The loss due to decentralization is thoroughly characterized. Local communication among sensors (using a sparsely connected graph), enables the on-line learning of unknown parameters of the statistical model. These learned parameters are employed to compute the desired thresholds locally and autonomously. Consensus-based strategies are investigated and analyzed for parameter estimation. One strategy approaches the performance of the decentralized approach with fast convergence and a second strategy approaches the performance of the centralized approach, albeit with slower convergence. A hybrid scheme that combines the best of both approaches is proposed offering a fast convergence and excellent convergent performance.
A theory for diffusivity estimation for spatially extended activator-inhibitor dynamics modelling the evolution of intracellular signaling networks is developed in the mathematical framework of stochastic reaction-diffusion systems. In order to account for model uncertainties, we extend the results for parameter estimation for semilinear stochastic partial differential equations, as developed in [PS20], to the problem of joint estimation of diffusivity and parametrized reaction terms. Our theoretical findings are applied to the estimation of effective diffusivity of signaling components contributing to intracellular dynamics of the actin cytoskeleton in the model organism Dictyostelium discoideum.