Low overhead channel estimation based on compressive sensing (CS) has been widely investigated for hybrid wideband millimeter wave (mmWave) multiple-input multiple-output (MIMO) systems. The channel sparsifying dictionaries used in prior work are built from ideal array response vectors evaluated on discrete angles of arrival/departure. In addition, these dictionaries are assumed to be the same for all subcarriers, without considering the impacts of hardware impairments and beam squint. In this manuscript, we derive a general channel and signal model that explicitly incorporates the impacts of hardware impairments, practical pulse shaping functions, and beam squint, overcoming the limitations of mmWave MIMO channel and signal models commonly used in previous work. Then, we propose a dictionary learning (DL) algorithm to obtain the sparsifying dictionaries embedding hardware impairments, by considering the effect of beam squint without introducing it into the learning process. We also design a novel CS channel estimation algorithm under beam squint and hardware impairments, where the channel structures at different subcarriers are exploited to enable channel parameter estimation with low complexity and high accuracy. Numerical results demonstrate the effectiveness of the proposed DL and channel estimation strategy when applied to realistic mmWave channels.
We propose a Web-Mashup Application Service Framework for Multivariate Time Series Analytics (MTSA) that supports the services of model definitions, querying, parameter learning, model evaluations, data monitoring, decision recommendations, and web portals. This framework maintains the advantage of combining the strengths of both the domain-knowledge-based and the formal-learning-based approaches and is designed for a more general class of problems over multivariate time series. More specifically, we identify a general-hybrid-based model, MTSA-Parameter Estimation, to solve this class of problems in which the objective function is maximized or minimized from the optimal decision parameters regardless of particular time points. This model also allows domain experts to include multiple types of constraints, e.g., global constraints and monitoring constraints. We further extend the MTSA data model and query language to support this class of problems for the services of learning, monitoring, and recommendation. At the end, we conduct an experimental case study for a university campus microgrid as a practical example to demonstrate our proposed framework, models, and language.
In an active sensing system, waveforms with good auto-correlations are preferred for accurate parameter estimation. Furthermore, spectral compatibility is required to avoid mutual interference between devices as the electromagnetic environment becomes increasingly crowded. Waveforms should also be unimodular due to hardware limits. In this paper, a new approach to generating a unimodular sequence with an approximately optimal peak side-lobe level (PSL) in auto-correlation and adjustable stopband attenuation is proposed. The proposed method is based on alternating minimization (AM) and numerical results suggest that it outperforms existing methods in terms of PSL. We also develop a theoretical lower bound for the PSL minimization problem under spectral constraints and unimodular constraints, which can be used for the evaluation of the results in various works about this waveform design problem. It is observed in the numerical results that the PSL of the proposed algorithm is close to the derived lower bound.
Parameter estimation and associated uncertainty quantification is an important problem in dynamical systems characterized by ordinary differential equation (ODE) models that are often nonlinear. Typically, such models have analytically intractable trajectories which result in likelihoods and posterior distributions that are similarly intractable. Bayesian inference for ODE systems via simulation methods require numerical approximations to produce inference with high accuracy at a cost of heavy computational power and slow convergence. At the same time, Artificial Neural Networks (ANN) offer tractability that can be utilized to construct an approximate but tractable likelihood and posterior distribution. In this paper we propose a hybrid approach, where Laplace-based Bayesian inference is combined with an ANN architecture for obtaining approximations to the ODE trajectories as a function of the unknown initial values and system parameters. Suitable choices of a collocation grid and customized loss functions are proposed to fine tune the ODE trajectories and Laplace approximation. The effectiveness of our proposed methods is demonstrated using an epidemiological system with non-analytical solutions, the Susceptible-Infectious-Removed (SIR) model for infectious diseases, based on simulated and real-life influenza datasets. The novelty and attractiveness of our proposed approach include (i) a new development of Bayesian inference using ANN architectures for ODE based dynamical systems, and (ii) a computationally fast posterior inference by avoiding convergence issues of benchmark Markov Chain Monte Carlo methods. These two features establish the developed approach as an accurate alternative to traditional Bayesian computational methods, with improved computational cost.
Symbolic recovery of differential equations is the ambitious attempt at automating the derivation of governing equations with the use of machine learning techniques. In contrast to classical methods which assume the structure of the equation to be known and focus on the estimation of specific parameters, these algorithms aim to learn the structure and the parameters simultaneously. While the uniqueness and, therefore, the identifiability of parameters of governing equations are a well-addressed problem in the field of parameter estimation, it has not been investigated for symbolic recovery. However, this problem should be even more present in this field since the algorithms aim to cover larger spaces of governing equations. In this paper, we investigate under which conditions a solution of a differential equation does not uniquely determine the equation itself. For various classes of differential equations, we provide both necessary and sufficient conditions for a function to uniquely determine the corresponding differential equation. We then use our results to devise numerical algorithms aiming to determine whether a function solves a differential equation uniquely. Finally, we provide extensive numerical experiments showing that our algorithms can indeed guarantee the uniqueness of the learned governing differential equation, without assuming any knowledge about the analytic form of function, thereby ensuring the reliability of the learned equation.
Regime shifts in high-dimensional time series arise naturally in many applications, from neuroimaging to finance. This problem has received considerable attention in low-dimensional settings, with both Bayesian and frequentist methods used extensively for parameter estimation. The EM algorithm is a particularly popular strategy for parameter estimation in low-dimensional settings, although the statistical properties of the resulting estimates have not been well understood. Furthermore, its extension to high-dimensional time series has proved challenging. To overcome these challenges, in this paper we propose an approximate EM algorithm for Markov-switching VAR models that leads to efficient computation and also facilitates the investigation of asymptotic properties of the resulting parameter estimates. We establish the consistency of the proposed EM algorithm in high dimensions and investigate its performance via simulation studies. We also demonstrate the algorithm by analyzing a brain electroencephalography (EEG) dataset recorded on a patient experiencing epileptic seizure.
Minimum Description Length (MDL) provides a framework and an objective for principled model evaluation. It formalizes Occam's Razor and can be applied to data from non-stationary sources. In the prequential formulation of MDL, the objective is to minimize the cumulative next-step log-loss when sequentially going through the data and using previous observations for parameter estimation. It thus closely resembles a continual- or online-learning problem. In this study, we evaluate approaches for computing prequential description lengths for image classification datasets with neural networks. Considering the computational cost, we find that online-learning with rehearsal has favorable performance compared to the previously widely used block-wise estimation. We propose forward-calibration to better align the models predictions with the empirical observations and introduce replay-streams, a minibatch incremental training technique to efficiently implement approximate random replay while avoiding large in-memory replay buffers. As a result, we present description lengths for a suite of image classification datasets that improve upon previously reported results by large margins.
This paper presents a data-driven, task-specific paradigm for experimental design, to shorten acquisition time, reduce costs, and accelerate the deployment of imaging devices. Current approaches in experimental design focus on model-parameter estimation and require specification of a particular model, whereas in imaging, other tasks may drive the design. Furthermore, such approaches often lead to intractable optimization problems in real-world imaging applications. Here we present a new paradigm for experimental design that simultaneously optimizes the design (set of image channels) and trains a machine-learning model to execute a user-specified image-analysis task. The approach obtains data densely-sampled over the measurement space (many image channels) for a small number of acquisitions, then identifies a subset of channels of prespecified size that best supports the task. We propose a method: TADRED for TAsk-DRiven Experimental Design in imaging, to identify the most informative channel-subset whilst simultaneously training a network to execute the task given the subset. Experiments demonstrate the potential of TADRED in diverse imaging applications: several clinically-relevant tasks in magnetic resonance imaging; and remote sensing and physiological applications of hyperspectral imaging. Results show substantial improvement over classical experimental design, two recent application-specific methods within the new paradigm, and state-of-the-art approaches in supervised feature selection. We anticipate further applications of our approach. Code is available: https://github.com/sbb-gh/experimental-design-multichannel
Homodyned K (HK) distribution has been widely used to describe the scattering phenomena arising in various research fields, such as ultrasound imaging or optics. In this work, we propose a machine learning based approach to the estimation of the HK distribution parameters. We develop neural networks that can estimate the HK distribution parameters based on the signal-to-noise ratio, skewness and kurtosis calculated using fractional-order moments. Compared to the previous approaches, we consider the orders of the moments as trainable variables that can be optimized along with the network weights using the back-propagation algorithm. Networks are trained based on samples generated from the HK distribution. Obtained results demonstrate that the proposed method can be used to accurately estimate the HK distribution parameters.
In this work, experiment design for marine vessels is explored. A dictionary-based approach is used, i.e., a systematic way of choosing the most informative combination of independent experiments out of a predefined set of candidates. This idea is quite general but is here tailored to an instrumental variable (IV) estimator with zero-mean instruments. This type of estimator is well-suited to deal with parameter estimation for second-order modulus models, which is a class of models often used to describe motion of marine vessels. The method is evaluated using both simulated and real data, the latter from a small model ship as well as from a full-scale vessel. Further, a standard motion-planning problem is modified to account for the prior-made choice of information-optimal sub-experiments, which makes it possible to obtain a plan for the complete experiment in the form of a feasible trajectory.
In this paper, we develop an active sensing strategy for a millimeter wave (mmWave) band Integrated Sensing and Communication (ISAC) system adopting a realistic hybrid digital-analog (HDA) architecture. To maintain a desired SNR level, initial beam acquisition (BA) must be established prior to data transmission. In the considered setup, a Base Station (BS) Tx transmits data via a digitally modulated waveform and a co-located radar receiver simultaneously performs radar estimation from the backscattered signal. In this BA scheme, a single common data stream is broadcast over a wide angular sector such that the radar receiver can detect the presence of not yet acquired users and perform coarse parameter estimation (angle of arrival, time of flight, and Doppler). As a result of the HDA architecture, we consider the design of multi-block adaptive RF-domain 'reduction matrices' (from antennas to RF chains) at the radar receiver, to achieve a compromise between the exploration capability in the angular domain and the directivity of the beamforming patterns. Our numerical results demonstrate that the proposed approach is able to reliably detect multiple targets while significantly reducing the initial acquisition time.
Quantifying the parameters and corresponding uncertainties of hundreds of strongly lensed quasar systems holds the key to resolving one of the most important scientific questions: the Hubble constant ($H_{0}$) tension. The commonly used Markov chain Monte Carlo (MCMC) method has been too time-consuming to achieve this goal, yet recent work has shown that convolution neural networks (CNNs) can be an alternative with seven orders of magnitude improvement in speed. With 31,200 simulated strongly lensed quasar images, we explore the usage of Vision Transformer (ViT) for simulated strong gravitational lensing for the first time. We show that ViT could reach competitive results compared with CNNs, and is specifically good at some lensing parameters, including the most important mass-related parameters such as the center of lens $\theta_{1}$ and $\theta_{2}$, the ellipticities $e_1$ and $e_2$, and the radial power-law slope $\gamma'$. With this promising preliminary result, we believe the ViT (or attention-based) network architecture can be an important tool for strong lensing science for the next generation of surveys. The open source of our code and data is in \url{https://github.com/kuanweih/strong_lensing_vit_resnet}.
Machine learning is a powerful approach for fitting microstructural models to diffusion MRI data. Early machine learning microstructure imaging implementations trained regressors to estimate model parameters in a supervised way, using synthetic training data with known ground truth. However, a drawback of this approach is that the choice of training data impacts fitted parameter values. Self-supervised learning is emerging as an attractive alternative to supervised learning in this context. Thus far, both supervised and self-supervised learning have typically been applied to isotropic models, such as intravoxel incoherent motion (IVIM), as opposed to models where the directionality of anisotropic structures is also estimated. In this paper, we demonstrate self-supervised machine learning model fitting for a directional microstructural model. In particular, we fit a combined T1-ball-stick model to the multidimensional diffusion (MUDI) challenge diffusion-relaxation dataset. Our self-supervised approach shows clear improvements in parameter estimation and computational time, for both simulated and in-vivo brain data, compared to standard non-linear least squares fitting. Code for the artificial neural net constructed for this study is available for public use from the following GitHub repository: https://github.com/jplte/deep-T1-ball-stick
The firing dynamics of biological neurons in mathematical models is often determined by the model's parameters, representing the neurons' underlying properties. The parameter estimation problem seeks to recover those parameters of a single neuron or a neuron population from their responses to external stimuli and interactions between themselves. Most common methods for tackling this problem in the literature use some mechanistic models in conjunction with either a simulation-based or solution-based optimization scheme. In this paper, we study an automatic approach of learning the parameters of neuron populations from a training set consisting of pairs of spiking series and parameter labels via supervised learning. Unlike previous work, this automatic learning does not require additional simulations at inference time nor expert knowledge in deriving an analytical solution or in constructing some approximate models. We simulate many neuronal populations with different parameter settings using a stochastic neuron model. Using that data, we train a variety of supervised machine learning models, including convolutional and deep neural networks, random forest, and support vector regression. We then compare their performance against classical approaches including a genetic search, Bayesian sequential estimation, and a random walk approximate model. The supervised models almost always outperform the classical methods in parameter estimation and spike reconstruction errors, and computation expense. Convolutional neural network, in particular, is the best among all models across all metrics. The supervised models can also generalize to out-of-distribution data to a certain extent.
Unsupervised learning has been widely used in many real-world applications. One of the simplest and most important unsupervised learning models is the Gaussian mixture model (GMM). In this work, we study the multi-task learning problem on GMMs, which aims to leverage potentially similar GMM parameter structures among tasks to obtain improved learning performance compared to single-task learning. We propose a multi-task GMM learning procedure based on the EM algorithm that effectively utilizes unknown similarities between related tasks and is robust against a fraction of outlier tasks from arbitrary distributions. The proposed procedure is shown to achieve the minimax optimal rate of convergence for both parameter estimation error and the excess mis-clustering error, in a wide range of regimes. Moreover, we generalize our approach to tackle the problem of transfer learning for GMMs, where similar theoretical results are derived. Additionally, iterative unsupervised multi-task and transfer learning methods may suffer from an initialization alignment problem, and two alignment algorithms are proposed to resolve the issue. Finally, we demonstrate the effectiveness of our methods through simulations and real data examples. To the best of our knowledge, this is the first work studying multi-task and transfer learning on GMMs with theoretical guarantees.
Causal inference and model interpretability research are gaining increasing attention, especially in the domains of healthcare and bioinformatics. Despite recent successes in this field, decorrelating features under nonlinear environments with human interpretable representations has not been adequately investigated. To address this issue, we introduce a novel method with a variable decorrelation regularizer to handle both linear and nonlinear confounding. Moreover, we employ association rules as new representations using association rule mining based on the original features to further proximate human decision patterns to increase model interpretability. Extensive experiments are conducted on four healthcare datasets (one synthetically generated and three real-world collections on different diseases). Quantitative results in comparison to baseline approaches on parameter estimation and causality computation indicate the model's superior performance. Furthermore, expert evaluation given by healthcare professionals validates the effectiveness and interpretability of the proposed model.
Several current ultra-wide band applications, such as millimeter wave radar and communication systems, require high sampling rates and therefore expensive and energy-hungry analogto-digital converters (ADCs). In applications where cost and power constraints exist, the use of high-precision ADCs is not feasible and the designer must resort to ADCs with coarse quantization. Consequently the interest in the topic of signal parameter estimation from quantized data has increased significantly in recent years. The Cramer-Rao bound (CRB) is an important yardstick in any parameter estimation problem. Indeed it lower bounds the variance of any unbiased parameter estimator. Moreover, the CRB is an achievable limit, for instance it is asymptotically attained by the maximum likelihood estimator (under regularity conditions), and thus it is a useful benchmark to which the accuracy of any parameter estimator can and should be compared. A formula for the CRB for signal parameter estimation from real-valued quantized data has been presented in but its derivation was somewhat sketchy. The said CRB formula has been extended for instance in to complex-valued quantized data, but again its derivation was rather sketchy. The special case of binary (1-bit) ADCs and a signal consisting of one sinusoid has been thoroughly analyzed in . The CRB formula for a binary ADC and a general real-valued signal has been derived.
Computational models have become a powerful tool in the quantitative sciences to understand the behaviour of complex systems that evolve in time. However, they often contain a potentially large number of free parameters whose values cannot be obtained from theory but need to be inferred from data. This is especially the case for models in the social sciences, economics, or computational epidemiology. Yet many current parameter estimation methods are mathematically involved and computationally slow to run. In this paper we present a computationally simple and fast method to retrieve accurate probability densities for model parameters using neural differential equations. We present a pipeline comprising multi-agent models acting as forward solvers for systems of ordinary or stochastic differential equations, and a neural network to then extract parameters from the data generated by the model. The two combined create a powerful tool that can quickly estimate densities on model parameters, even for very large systems. We demonstrate the method on synthetic time series data of the SIR model of the spread of infection, and perform an in-depth analysis of the Harris-Wilson model of economic activity on a network, representing a non-convex problem. For the latter, we apply our method both to synthetic data and to data of economic activity across Greater London. We find that our method calibrates the model orders of magnitude more accurately than a previous study of the same dataset using classical techniques, while running between 195 and 390 times faster.
This paper proposes an adaptive tracking strategy with mass-inertia estimation for aerial transportation problems of multi-rotor UAVs. The dynamic model of multi-rotor UAVs with disturbances is firstly developed with a linearly parameterized form. Subsequently, a cascade controller with the robust integral of the sign of the error (RISE) terms is applied to smooth the control inputs and address bounded disturbances. Then, adaptive estimation laws for mass-inertia parameters are designed based on a filter operation. Such operation is introduced to extract estimation errors exploited to theoretically guarantee the finite-time (FT) convergence of estimation errors. Finally, simulations are conducted to verify the effectiveness of the designed controller. The results show that the proposed method provides better tracking and estimation performance than traditional adaptive controllers based on sliding mode control algorithms and gradient-based estimation strategies.
Federated learning, where algorithms are trained across multiple decentralized devices without sharing local data, is increasingly popular in distributed machine learning practice. Typically, a graph structure $G$ exists behind local devices for communication. In this work, we consider parameter estimation in federated learning with data distribution and communication heterogeneity, as well as limited computational capacity of local devices. We encode the distribution heterogeneity by parametrizing distributions on local devices with a set of distinct $p$-dimensional vectors. We then propose to jointly estimate parameters of all devices under the $M$-estimation framework with the fused Lasso regularization, encouraging an equal estimate of parameters on connected devices in $G$. We provide a general result for our estimator depending on $G$, which can be further calibrated to obtain convergence rates for various specific problem setups. Surprisingly, our estimator attains the optimal rate under certain graph fidelity condition on $G$, as if we could aggregate all samples sharing the same distribution. If the graph fidelity condition is not met, we propose an edge selection procedure via multiple testing to ensure the optimality. To ease the burden of local computation, a decentralized stochastic version of ADMM is provided, with convergence rate $O(T^{-1}\log T)$ where $T$ denotes the number of iterations. We highlight that, our algorithm transmits only parameters along edges of $G$ at each iteration, without requiring a central machine, which preserves privacy. We further extend it to the case where devices are randomly inaccessible during the training process, with a similar algorithmic convergence guarantee. The computational and statistical efficiency of our method is evidenced by simulation experiments and the 2020 US presidential election data set.