Phase-locked loops (PLLs) are now widely used in communication systems and have been a classic system for more than 60 years. Well-known mathematical models of such systems are constructed in a number of approximations, so questions about how they describe the experimental dynamics qualitatively and quantitatively, and how the accuracy of the model description depends on the behavior mode, remain open. One of the most direct approaches to the verification of any model is its reconstruction from the time series obtained in experiment. If it is possible to fit the model to experimental data and the resulting parameter values are close to the expected values (calculated from the first principles), the quantitative correspondence between the model and the physical object is nearly proved. In this paper, for the first time, the equations of the PLL model with a bandpass filter are reconstructed from the experimental signals of the generator in various modes. The reconstruction showed that the model known in the literature generally describes the experimental dynamics in regular and chaotic regimes. The relative error of parameter estimation is between 2% and 50% for different regimes and parameters. The reconstructed nonlinear function of phase is not harmonic and highly asymmetric in contrary to the model one.
The integration of renewables into electrical grids calls for optimization-based control schemes requiring reliable grid models. Classically, parameter estimation and optimization-based control is often decoupled, which leads to high system operation cost in the estimation procedure. The present work proposes a method for simultaneously minimizing grid operation cost and optimally estimating line parameters based on methods for the optimal design of experiments. This method leads to a substantial reduction in cost for optimal estimation and in higher accuracy in the parameters compared with standard Optimal Power Flow and maximum-likelihood estimation. We illustrate the performance of the proposed method on a benchmark system.
The time division multiple access (TDMA) technique has been applied in automotive multiple-input multiple-output (MIMO) radar. However, it suffers from the transmit energy loss, and as a result the parameter estimation performance degradation when the number of transmit elements increases. To tackle these problem, a transmit beamspace (TB) Doppler division multiple access (DDMA) approach is proposed. First, a phase modulation matrix with empty Doppler spectrum is introduced. By exploiting the empty Doppler spectrum, a test function based on sequential detection is developed to mitigate the Doppler ambiguity in DDMA waveform. Then, a discrete Fourier transform (DFT)-based TB in slow-time is formed.The proposed method can achieve waveform diversity in Doppler domain and generate a TB in slow-time that concentrates the transmitted power in a fixed spatial region to improve the transmit energy distribution for automotive MIMO radar, which is favored by medium/long range radar (MRR/LRR) applications. As compared to the conventional TDMA technique, the proposed TB DDMA approach can fully exploit the transmission capabilities of all transmit elements to ensure that the emitted power is efficiently used and inherits easy implementation. Moreover, the proposed TB DDMA method avoids the trade-off between the active time for each transmit antenna and the frame time. Simulation results verify the effectiveness of the proposed TB DDMA approach for automotive MIMO radar.
Mixture Density Networks (MDNs) can be used to generate probability density functions of model parameters $\boldsymbol{\theta}$ given a set of observables $\mathbf{x}$. In some applications, training data are available only for discrete values of a continuous parameter $\boldsymbol{\theta}$. In such situations a number of performance-limiting issues arise which can result in biased estimates. We demonstrate the usage of MDNs for parameter estimation, discuss the origins of the biases, and propose a corrective method for each issue.
Radio frequency interference (RFI) mitigation and radar echo recovery are critically important for the proper functioning of ultra-wideband (UWB) radar systems using one-bit sampling techniques. We recently introduced a technique for one-bit UWB radar, which first uses a majorization-minimization method for RFI parameter estimation followed by a sparse method for radar echo recovery. However, this technique suffers from high computational complexity due to the need to estimate the parameters of each RFI source separately and iteratively. In this paper, we present a computationally efficient joint RFI mitigation and radar echo recovery framework to greatly reduce the computational cost. Specifically, we exploit the sparsity of RFI in the fast-frequency domain and the sparsity of radar echoes in the fast-time domain to design a one-bit weighted SPICE (SParse Iterative Covariance-based Estimation) based framework for the joint RFI mitigation and radar echo recovery of one-bit UWB radar. Both simulated and experimental results are presented to show that the proposed one-bit weighted SPICE framework can not only reduce the computational cost but also outperform the existing approach for decoupled RFI mitigation and radar echo recovery of one-bit UWB radar.
In this work we demonstrate the use of neural networks for rapid extraction of signal parameters of discretely sampled signals. In particular, we use dense autoencoder networks to extract the parameters of interest from exponentially decaying signals and decaying oscillations. By using a three-stage training method and careful choice of the neural network size, we are able to retrieve the relevant signal parameters directly from the latent space of the autoencoder network at significantly improved rates compared to traditional algorithmic signal-analysis approaches. We show that the achievable precision and accuracy of this method of analysis is similar to conventional algorithm-based signal analysis methods, by demonstrating that the extracted signal parameters are approaching their fundamental parameter estimation limit as provided by the Cram\'er-Rao bound. Furthermore, we demonstrate that autoencoder networks are able to achieve signal analysis, and, hence, parameter extraction, at rates of 75 kHz, orders-of-magnitude faster than conventional techniques with similar precision. Finally, we explore the limitations of our approach, demonstrating that analysis rates of $>$200 kHz are feasible with further optimization of the transfer rate between the data-acquisition system and data-analysis system.
Motivated by recent studies in the data mining community which require to efficiently list all k-cliques, we revisit the iconic algorithm of Chiba and Nishizeki and develop the most efficient parallel algorithm for such a problem. Our theoretical analysis provides the best asymptotic upper bound on the running time of our algorithm for the case when the input graph is sparse. Our experimental evaluation on large real-world graphs shows that our parallel algorithm is faster than state-of-the-art algorithms, while boasting an excellent degree of parallelism. In particular, we are able to list all k-cliques (for any k) in graphs containing up to tens of millions of edges as well as all $10$-cliques in graphs containing billions of edges, within a few minutes and a few hours respectively. Finally, we show how our algorithm can be employed as an effective subroutine for finding the k-clique core decomposition and an approximate k-clique densest subgraphs in very large real-world graphs.
Purpose: Undersampling is used to reduce the scan time for high-resolution 3D magnetic resonance imaging. In order to achieve better image quality and avoid manual parameter tuning, we propose a probabilistic Bayesian approach to recover $R_2^*$ map and phase images for quantitative susceptibility mapping (QSM), while allowing automatic parameter estimation from undersampled data. Theory: Sparse prior on the wavelet coefficients of images is interpreted from a Bayesian perspective as sparsity-promoting distribution. A novel nonlinear approximate message passing (AMP) framework that incorporates a mono-exponential decay model is proposed. The parameters are treated as unknown variables and jointly estimated with image wavelet coefficients. Results: The proposed AMP with parameter estimation (AMP-PE) approach successfully recovers $R_2^*$ maps and phase images for QSM across various undersampling rates. It is more computationally efficient, and performs better than the state-of-the-art $l_1$-norm regularization (L1) approach in general, except a few cases where the L1 approach performs as well as AMP-PE. Conclusion: AMP-PE achieves better performance by drawing information from both the sparse prior and the mono-exponential decay model. It does not require parameter tuning, and works with a clinical, prospective undersampling scheme where parameter tuning is often impossible or difficult due to the lack of ground-truth image.
Hyperspectral image (HSI) super-resolution is commonly used to overcome the hardware limitations of existing hyperspectral imaging systems on spatial resolution. It fuses a low-resolution (LR) HSI and a high-resolution (HR) conventional image of the same scene to obtain an HR HSI. In this work, we propose a method that integrates a physical model and deep prior information. Specifically, a novel, yet effective two-stream fusion network is designed to serve as a {regularizer} for the fusion problem. This fusion problem is formulated as an optimization problem whose solution can be obtained by solving a Sylvester equation. Furthermore, the regularization parameter is simultaneously estimated to automatically adjust contribution of the physical model and {the} learned prior to reconstruct the final HR HSI. Experimental results on {both simulated and real data} demonstrate the superiority of the proposed method over other state-of-the-art methods on both quantitative and qualitative comparisons.
We present an approach for state and parameter estimation in retinal laser treatment by a novel setup where both measurement and heating is performed by a single laser. In this medical application, the temperature that is induced by the laser in the patient's eye is critical for a successful and safe treatment. To this end, we pursue a model-based approach using a model given by a heat diffusion equation on a cylindrical domain, where the source term is given by the absorbed laser power. The model is parametric in the sense that it involves an absorption coefficient, which depends on the treatment spot and plays a central role in the input-output behavior of the system. After discretization, we apply a particularly suited parametric model order reduction to ensure real-time tractability while retaining parameter dependence. We augment known state estimation techniques, i.e., extended Kalman filtering and moving horizon estimation, with parameter estimation to estimate the absorption coefficient and the current state of the system. Eventually, we show first results for simulated and experimental data from porcine eyes. We find that, regarding convergence speed, the moving horizon estimation slightly outperforms the extended Kalman filter on measurement data in terms of parameter and state estimation, however, on simulated data the results are very similar.
This study applies convolutional neural network (CNN)-based automatic segmentation and distensibility measurement of the ascending and descending aorta from 2D phase-contrast cine magnetic resonance imaging (PC-cine MRI) within the large MESA cohort with subsequent assessment on an external cohort of thoracic aortic aneurysm (TAA) patients. 2D PC-cine MRI images of the ascending and descending aorta at the pulmonary artery bifurcation from the MESA study were included. Train, validation, and internal test sets consisted of 1123 studies (24282 images), 374 studies (8067 images), and 375 studies (8069 images), respectively. An external test set of TAAs consisted of 37 studies (3224 images). A U-Net based CNN was constructed, and performance was evaluated utilizing dice coefficient (for segmentation) and concordance correlation coefficients (CCC) of aortic geometric parameters by comparing to manual segmentation and parameter estimation. Dice coefficients for aorta segmentation were 97.6% (CI: 97.5%-97.6%) and 93.6% (84.6%-96.7%) on the internal and external test of TAAs, respectively. CCC for comparison of manual and CNN maximum and minimum ascending aortic areas were 0.97 and 0.95, respectively, on the internal test set and 0.997 and 0.995, respectively, for the external test. CCCs for maximum and minimum descending aortic areas were 0.96 and 0. 98, respectively, on the internal test set and 0.93 and 0.93, respectively, on the external test set. We successfully developed and validated a U-Net based ascending and descending aortic segmentation and distensibility quantification model in a large multi-ethnic database and in an external cohort of TAA patients.
Dynamic causal modeling (DCM) is a Bayesian framework to infer directed connections between compartments, and has been used to describe the interactions between underlying neural populations based on functional neuroimaging data. DCM is typically analyzed with the expectation-maximization (EM) algorithm. However, because the inversion of a large-scale continuous system is difficult when noisy observations are present, DCM by EM is typically limited to a small number of compartments ($<10$). Another drawback with the current method is its complexity; when the forward model changes, the posterior mean changes, and we need to re-derive the algorithm for optimization. In this project, we propose the Multiple-Shooting Adjoint (MSA) method to address these limitations. MSA uses the multiple-shooting method for parameter estimation in ordinary differential equations (ODEs) under noisy observations, and is suitable for large-scale systems such as whole-brain analysis in functional MRI (fMRI). Furthermore, MSA uses the adjoint method for accurate gradient estimation in the ODE; since the adjoint method is generic, MSA is a generic method for both linear and non-linear systems, and does not require re-derivation of the algorithm as in EM. We validate MSA in extensive experiments: 1) in toy examples with both linear and non-linear models, we show that MSA achieves better accuracy in parameter value estimation than EM; furthermore, MSA can be successfully applied to large systems with up to 100 compartments; and 2) using real fMRI data, we apply MSA to the estimation of the whole-brain effective connectome and show improved classification of autism spectrum disorder (ASD) vs. control compared to using the functional connectome. The package is provided \url{https://jzkay12.github.io/TorchDiffEqPack}
Robust parameter estimation is a crucial task in several 3D computer vision pipelines such as Structure from Motion (SfM). State-of-the-art algorithms for robust estimation, however, still suffer from difficulties in converging to satisfactory solutions due to the presence of many poor local minima or flat regions in the optimization landscapes. In this paper, we introduce two novel approaches for robust parameter estimation. The first algorithm utilizes the Filter Method (FM), which is a framework for constrained optimization allowing great flexibility in algorithmic choices, to derive an adaptive kernel scaling strategy that enjoys a strong ability to escape poor minima and achieves fast convergence rates. Our second algorithm combines a generalized Majorization Minimization (GeMM) framework with the half-quadratic lifting formulation to obtain a simple yet efficient solver for robust estimation. We empirically show that both proposed approaches show encouraging capability on avoiding poor local minima and achieve competitive results compared to existing state-of-the art robust fitting algorithms.
We present the framework of slowly varying regression under sparsity, allowing sparse regression models to exhibit slow and sparse variations. The problem of parameter estimation is formulated as a mixed-integer optimization problem. We demonstrate that it can be precisely reformulated as a binary convex optimization problem through a novel relaxation technique. This relaxation involves a new equality on Moore-Penrose inverses, convexifying the non-convex objective function while matching the original objective on all feasible binary points. This enables us to efficiently solve the problem to provable optimality using a cutting plane-type algorithm. We develop a highly optimized implementation of this algorithm, substantially improving upon the asymptotic computational complexity of a straightforward implementation. Additionally, we propose a fast heuristic method that guarantees a feasible solution and, as empirically illustrated, produces high-quality warm-start solutions for the binary optimization problem. To tune the framework's hyperparameters, we suggest a practical procedure relying on binary search that, under certain assumptions, is guaranteed to recover the true model parameters. On both synthetic and real-world datasets, we demonstrate that the resulting algorithm outperforms competing formulations in comparable times across various metrics, including estimation accuracy, predictive power, and computational time. The algorithm is highly scalable, allowing us to train models with thousands of parameters. Our implementation is available open-source at https://github.com/vvdigalakis/SSVRegression.git.
Stochastic high dimensional bandit problems with low dimensional structures are useful in different applications such as online advertising and drug discovery. In this work, we propose a simple unified algorithm for such problems and present a general analysis framework for the regret upper bound of our algorithm. We show that under some mild unified assumptions, our algorithm can be applied to different high dimensional bandit problems. Our framework utilizes the low dimensional structure to guide the parameter estimation in the problem, therefore our algorithm achieves the comparable regret bounds in the LASSO bandit, as well as novel bounds in the low-rank matrix bandit, the group sparse matrix bandit, and in a new problem: the multi-agent LASSO bandit.
Logistic regression is a commonly used method for binary classification. Researchers often have more than a single binary response variable and simultaneous analysis is beneficial because it provides insight into the dependencies among response variables as well as between the predictor variables and the responses. Moreover, in such a simultaneous analysis the equations can lend each other strength, which might increase predictive accuracy. In this paper, we propose the MELODIC family for simultaneous binary logistic regression modeling. In this family, the regression models are defined in a Euclidean space of reduced dimension, based on a distance rule. The model may be interpreted in terms of logistic regression coefficients or in terms of a biplot. We discuss a fast iterative majorization (or MM) algorithm for parameter estimation. Two applications are shown in detail: one relating personality characteristics to drug consumption profiles and one relating personality characteristics to depressive and anxiety disorders. We present a thorough comparison of our MELODIC family with alternative approaches for multivariate binary data.
Considering the problem of novel view synthesis (NVS) from only a set of 2D images, we simplify the training process of Neural Radiance Field (NeRF) on forward-facing scenes by removing the requirement of known or pre-computed camera parameters, including both intrinsics and 6DoF poses. To this end, we propose NeRF$--$, with three contributions: First, we show that the camera parameters can be jointly optimised as learnable parameters with NeRF training, through a photometric reconstruction; Second, to benchmark the camera parameter estimation and the quality of novel view renderings, we introduce a new dataset of path-traced synthetic scenes, termed as Blender Forward-Facing Dataset (BLEFF); Third, we conduct extensive analyses to understand the training behaviours under various camera motions, and show that in most scenarios, the joint optimisation pipeline can recover accurate camera parameters and achieve comparable novel view synthesis quality as those trained with COLMAP pre-computed camera parameters. Our code and data are available at https://nerfmm.active.vision.
Parameter Estimation (PE) and State Estimation (SE) are the most wide-spread tasks in the system engineering. They need to be done automatically, fast and frequently, as measurements arrive. Deep Learning (DL) holds the promise of tackling the challenge, however in so far, as PE and SE in power systems is concerned, (a) DL did not win trust of the system operators because of the lack of the physics of electricity based, interpretations and (b) DL remained illusive in the operational regimes were data is scarce. To address this, we present a hybrid scheme which embeds physics modeling of power systems into Graphical Neural Networks (GNN), therefore empowering system operators with a reliable and explainable real-time predictions which can then be used to control the critical infrastructure. To enable progress towards trustworthy DL for PE and SE, we build a physics-informed method, named Power-GNN, which reconstructs physical, thus interpretable, parameters within Effective Power Flow (EPF) models, such as admittances of effective power lines, and NN parameters, representing implicitly unobserved elements of the system. In our experiments, we test the Power-GNN on different realistic power networks, including these with thousands of loads and hundreds of generators. We show that the Power-GNN outperforms vanilla NN scheme unaware of the EPF physics.
Statistical machine learning models trained with stochastic gradient algorithms are increasingly being deployed in critical scientific applications. However, computing the stochastic gradient in several such applications is highly expensive or even impossible at times. In such cases, derivative-free or zeroth-order algorithms are used. An important question which has thus far not been addressed sufficiently in the statistical machine learning literature is that of equipping stochastic zeroth-order algorithms with practical yet rigorous inferential capabilities so that we not only have point estimates or predictions but also quantify the associated uncertainty via confidence intervals or sets. Towards this, in this work, we first establish a central limit theorem for Polyak-Ruppert averaged stochastic zeroth-order gradient algorithm. We then provide online estimators of the asymptotic covariance matrix appearing in the central limit theorem, thereby providing a practical procedure for constructing asymptotically valid confidence sets (or intervals) for parameter estimation (or prediction) in the zeroth-order setting.
Estimation of spatially-varying parameters for computationally expensive forward models governed by partial differential equations is addressed. A novel multiscale Bayesian inference approach is introduced based on deep probabilistic generative models. Such generative models provide a flexible representation by inferring on each scale a low-dimensional latent encoding while allowing hierarchical parameter generation from coarse- to fine-scales. Combining the multiscale generative model with Markov Chain Monte Carlo (MCMC), inference across scales is achieved enabling us to efficiently obtain posterior parameter samples at various scales. The estimation of coarse-scale parameters using a low-dimensional latent embedding captures global and notable parameter features using an inexpensive but inaccurate solver. MCMC sampling of the fine-scale parameters is enabled by utilizing the posterior information in the immediate coarser-scale. In this way, the global features are identified in the coarse-scale with inference of low-dimensional variables and inexpensive forward computation, and the local features are refined and corrected in the fine-scale. The developed method is demonstrated with two types of permeability estimation for flow in heterogeneous media. One is a Gaussian random field (GRF) with uncertain length scales, and the other is channelized permeability with the two regions defined by different GRFs. The obtained results indicate that the method allows high-dimensional parameter estimation while exhibiting stability, efficiency and accuracy.