Temporal-Difference (TD) learning methods, such as Q-Learning, have proven effective at learning a policy to perform control tasks. One issue with methods like Q-Learning is that the value update introduces bias when predicting the TD target of a unfamiliar state. Estimation noise becomes a bias after the max operator in the policy improvement step, and carries over to value estimations of other states, causing Q-Learning to overestimate the Q value. Algorithms like Soft Q-Learning (SQL) introduce the notion of a soft-greedy policy, which reduces the estimation bias via soft updates in early stages of training. However, the inverse temperature $\beta$ that controls the softness of an update is usually set by a hand-designed heuristic, which can be inaccurate at capturing the uncertainty in the target estimate. Under the belief that $\beta$ is closely related to the (state dependent) model uncertainty, Entropy Regularized Q-Learning (EQL) further introduces a principled scheduling of $\beta$ by maintaining a collection of the model parameters that characterizes model uncertainty. In this paper, we present Unbiased Soft Q-Learning (UQL), which extends the work of EQL from two action, finite state spaces to multi-action, infinite state space Markov Decision Processes. We also provide a principled numerical scheduling of $\beta$, extended from SQL and using model uncertainty, during the optimization process. We show the theoretical guarantees and the effectiveness of this update method in experiments on several discrete control environments.
Lithium-ion battery degradation estimation using fast and noninvasive techniques is a crucial issue in the circular economy framework of this technology. Currently, most of the approaches used to establish the battery-state (i.e., State of Charge (SoC), State of Health (SoH)) require time-consuming processes. In the present preliminary study, an ultrasound array was used to assess the influence of the SoC and SoH on the variations in the time of flight (TOF) and the speed of sound (SOS) of the ultrasound wave inside the batteries. Nine aged 18650 Lithium-ion batteries were imaged at 100% and 0% SoC using a Vantage-256 system (Verasonics, Inc.) equipped with a 64-element ultrasound array and a center frequency of 5 MHz (Imasonic SAS). It was found that second-life batteries have a complex ultrasound response due to the presence of many degradation pathways and, thus, making it harder to analyze the ultrasound measurements. Although further analysis must be done to elucidate a clear correlation between changes in the ultrasound wave properties and the battery state estimation, this approach seems very promising for future nondestructive evaluation of second-life batteries.
We present a variational method for online state estimation and parameter learning in state-space models (SSMs), a ubiquitous class of latent variable models for sequential data. As per standard batch variational techniques, we use stochastic gradients to simultaneously optimize a lower bound on the log evidence with respect to both model parameters and a variational approximation of the states' posterior distribution. However, unlike existing approaches, our method is able to operate in an entirely online manner, such that historic observations do not require revisitation after being incorporated and the cost of updates at each time step remains constant, despite the growing dimensionality of the joint posterior distribution of the states. This is achieved by utilizing backward decompositions of this joint posterior distribution and of its variational approximation, combined with Bellman-type recursions for the evidence lower bound and its gradients. We demonstrate the performance of this methodology across several examples, including high-dimensional SSMs and sequential Variational Auto-Encoders.
The robust distributed state estimation for a class of continuous-time linear time-invariant systems is achieved by a novel kernel-based distributed observer, which, for the first time, ensures fixed-time convergence properties. The communication network between the agents is prescribed by a directed graph in which each node involves a fixed-time convergent estimator. The local observer estimates and broadcasts the observable states among neighbours so that the full state vector can be recovered at each node and the estimation error reaches zero after a predefined fixed time in the absence of perturbation. This represents a new distributed estimation framework that enables faster convergence speed and further reduced information exchange compared to a conventional Luenberger-like approach. The ubiquitous time-varying communication delay across the network is suitably compensated by a prediction scheme. Moreover, the robustness of the algorithm in the presence of bounded measurement and process noise is characterised. Numerical simulations and comparisons demonstrate the effectiveness of the observer and its advantages over the existing methods.
Conventionally, the dynamic state estimation of variables in power networks is performed based on the forecasting-aided model of bus voltages. This approach is effective in the stiff grids at the transmission level, where the bus voltages are less sensitive to variations of the load. However, in microgrids, bus voltages can fluctuate significantly under load changes, the forecasting-aided model may not sufficiently accurate. To resolve this problem, this paper proposes a dynamic state estimation scheme for microgrids using the state-space model derived from differential equations of power networks. In the proposed scheme, the branch currents are the state variables, whereas the bus voltages become the inputs which can vary freely with loads. As a result, the entire microgrids system can be partitioned into local areas, where neighbor areas share the common inputs. The proposed estimation scheme then can be implemented in a distributed manner. A novel Kalman-based filtering method is derived to estimate both states and inputs simultaneously. Only information of common inputs is exchanged between neighboring estimators. Simulation results of the 13-bus Potsdam microgrid (New York State) are provided to prove the feasibility and performances of the proposed scheme.
While recent years have witnessed remarkable progress in the feature representation of visual tracking, the problem of feature misalignment between the classification and regression tasks is largely overlooked. The approaches of feature extraction make no difference for these two tasks in most of advanced trackers. We argue that the performance gain of visual tracking is limited since features extracted from the salient area provide more recognizable visual patterns for classification, while these around the boundaries contribute to accurately estimating the target state. We address this problem by proposing two customized feature extractors, named polar pooling and extreme pooling to capture task-specific visual patterns. Polar pooling plays the role of enriching information collected from the semantic keypoints for stronger classification, while extreme pooling facilitates explicit visual patterns of the object boundary for accurate target state estimation. We demonstrate the effectiveness of the task-specific feature representation by integrating it into the recent and advanced tracker RPT. Extensive experiments on several benchmarks show that our Customized Features based RPT (RPT++) achieves new state-of-the-art performances on OTB-100, VOT2018, VOT2019, GOT-10k, TrackingNet and LaSOT.
Consistent state estimation is challenging, especially under the epistemic uncertainties arising from learned (nonlinear) dynamic and observation models. In this work, we propose a set-based estimation algorithm, named Gaussian Process-Zonotopic Kalman Filter (GP-ZKF), that produces zonotopic state estimates while respecting both the epistemic uncertainties in the learned models and aleatoric uncertainties. Our method guarantees probabilistic consistency, in the sense that the true states are bounded by sets (zonotopes) across all time steps, with high probability. We formally relate GP-ZKF with the corresponding stochastic approach, GP-EKF, in the case of learned (nonlinear) models. In particular, when linearization errors and aleatoric uncertainties are omitted and epistemic uncertainties are simplified, GP-ZKF reduces to GP-EKF. We empirically demonstrate our method's efficacy in both a simulated pendulum domain and a real-world robot-assisted dressing domain, where GP-ZKF produced more consistent and less conservative set-based estimates than all baseline stochastic methods.
In this paper, an attack problem is investigated for event-based remote state estimation in cyber-physical systems. Our objective is to degrade the effect of the event-based scheduler while bypassing a $\chi^2$ false data detector. A two-channel scheduler-pointed false data injection attack strategy is proposed by modifying the numerical characteristics of innovation signals. The attack strategy is proved to be always existent, and an algorithm is provided to find it. Under the proposed attack strategy, the scheduler becomes almost invalid and the performance of the remote estimator is degraded. Numerical simulations are used to illustrate our theoretical results.
Many time series can be modeled as a sequence of segments representing high-level discrete states, such as running and walking in a human activity application. Flexible models should describe the system state and observations in stationary "pure-state" periods as well as transition periods between adjacent segments, such as a gradual slowdown between running and walking. However, most prior work assumes instantaneous transitions between pure discrete states. We propose a dynamical Wasserstein barycentric (DWB) model that estimates the system state over time as well as the data-generating distributions of pure states in an unsupervised manner. Our model assumes each pure state generates data from a multivariate normal distribution, and characterizes transitions between states via displacement-interpolation specified by the Wasserstein barycenter. The system state is represented by a barycentric weight vector which evolves over time via a random walk on the simplex. Parameter learning leverages the natural Riemannian geometry of Gaussian distributions under the Wasserstein distance, which leads to improved convergence speeds. Experiments on several human activity datasets show that our proposed DWB model accurately learns the generating distribution of pure states while improving state estimation for transition periods compared to the commonly used linear interpolation mixture models.
Modeling biological dynamical systems is challenging due to the interdependence of different system components, some of which are not fully understood. To fill existing gaps in our ability to mechanistically model physiological systems, we propose to combine neural networks with physics-based models. Specifically, we demonstrate how we can approximate missing ordinary differential equations (ODEs) coupled with known ODEs using Bayesian filtering techniques to train the model parameters and simultaneously estimate dynamic state variables. As a study case we leverage a well-understood model for blood circulation in the human retina and replace one of its core ODEs with a neural network approximation, representing the case where we have incomplete knowledge of the physiological state dynamics. Results demonstrate that state dynamics corresponding to the missing ODEs can be approximated well using a neural network trained using a recursive Bayesian filtering approach in a fashion coupled with the known state dynamic differential equations. This demonstrates that dynamics and impact of missing state variables can be captured through joint state estimation and model parameter estimation within a recursive Bayesian state estimation (RBSE) framework. Results also indicate that this RBSE approach to training the NN parameters yields better outcomes (measurement/state estimation accuracy) than training the neural network with backpropagation through time in the same setting.
This paper addresses state estimation of linear systems with special attention on unknown process and measurement noise covariances, aiming to enhance estimation accuracy while preserving the stability guarantee of the Kalman filter. To this end, the full information estimation problem over a finite interval is firstly addressed. Then, a novel adaptive variational Bayesian (VB) moving horizon estimation (MHE) method is proposed, exploiting VB inference, MHE and Monte Carlo integration with importance sampling for joint estimation of the unknown process and measurement noise covariances, along with the state trajectory over a moving window of fixed length. Further, it is proved that the proposed adaptive VB MHE filter ensures mean-square boundedness of the estimation error with any number of importance samples and VB iterations, as well as for any window length. Finally, simulation results on a target tracking example demonstrate the effectiveness of the VB MHE filter with enhanced estimation accuracy and convergence properties compared to the conventional non-adaptive Kalman filter and other existing adaptive filters.
State estimation for a class of linear time-invariant systems with distributed output measurements (distributed sensors) and unknown inputs is addressed in this paper. The objective is to design a network of observers such that the state vector of the entire system can be estimated, while each observer (or node) has access to only local output measurements that may not be sufficient on its own to reconstruct the entire system's state. Existing results in the literature on distributed state estimation assume either that the system does not have inputs, or that all the system's inputs are globally known to all the observers. Accordingly, we address this gap by proposing a distributed observer capable of estimating the overall system's state in the presence of inputs, while each node only has limited local information on inputs and outputs. We provide a design method that guarantees convergence of the estimation errors under some mild joint detectability conditions. This design suits undirected communication graphs that may have switching topologies and also applies to strongly connected directed graphs. We also give existence conditions that harmonize with existing results on unknown input observers. Finally, simulation results verify the effectiveness of the proposed estimation scheme for various scenarios.
The ongoing electrification introduces new challenges to distribution system operators (DSOs). Controllable resources may simultaneously react to price signals, potentially leading to network violations. DSOs require reliable and accurate low-voltage state estimation (LVSE) to improve awareness and mitigate such events. However, the influence of flexibility activations on LVSE has not been addressed yet. It remains unclear if flexibility-induced uncertainty can be reliably quantified to enable robust DSO decision-making. In this work, uncertainty quantification in LVSE is systematically investigated for multiple scenarios of input availability and flexibility utilization, using real data. For that purpose, a Bayesian neural network (BNN) is compared to quantile regression. Results show that frequent flexibility activations can significantly deteriorate LVSE performance, unless secondary substation measurements are available. Moreover, it is demonstrated that the BNN captures flexibility-induced voltage drops by dynamically extending the prediction interval during activation periods, and that it improves interpretability regarding the cause of uncertainty.
Exploiting past 3D LiDAR scans to predict future point clouds is a promising method for autonomous mobile systems to realize foresighted state estimation, collision avoidance, and planning. In this paper, we address the problem of predicting future 3D LiDAR point clouds given a sequence of past LiDAR scans. Estimating the future scene on the sensor level does not require any preceding steps as in localization or tracking systems and can be trained self-supervised. We propose an end-to-end approach that exploits a 2D range image representation of each 3D LiDAR scan and concatenates a sequence of range images to obtain a 3D tensor. Based on such tensors, we develop an encoder-decoder architecture using 3D convolutions to jointly aggregate spatial and temporal information of the scene and to predict the future 3D point clouds. We evaluate our method on multiple datasets and the experimental results suggest that our method outperforms existing point cloud prediction architectures and generalizes well to new, unseen environments without additional fine-tuning. Our method operates online and is faster than the common LiDAR frame rate of 10 Hz.
This work addresses the problem of sensing the world: how to learn a multimodal representation of a reinforcement learning agent's environment that allows the execution of tasks under incomplete perceptual conditions. To address such problem, we argue for hierarchy in the design of representation models and contribute with a novel multimodal representation model, MUSE. The proposed model learns hierarchical representations: low-level modality-specific representations, encoded from raw observation data, and a high-level multimodal representation, encoding joint-modality information to allow robust state estimation. We employ MUSE as the sensory representation model of deep reinforcement learning agents provided with multimodal observations in Atari games. We perform a comparative study over different designs of reinforcement learning agents, showing that MUSE allows agents to perform tasks under incomplete perceptual experience with minimal performance loss. Finally, we evaluate the performance of MUSE in literature-standard multimodal scenarios with higher number and more complex modalities, showing that it outperforms state-of-the-art multimodal variational autoencoders in single and cross-modality generation.
In this article, we present a new methodology to identify if forced oscillation sources are generated in excitation systems or in turbine governors. In this context, we propose to harness the information that the dynamic state estimation (DSE) provide to be able to apply a dissipating energy flow (DEF) method. The measurements that the phasor measurement units (PMUs) provide are used to estimate the internal states of different generators connected to the same bus. Here, it will be considered that the PMU at the point of connection is only capable of measuring the voltage phasor and the total current phasor. Therefore, the current injection of the generator that is causing the FO is an unobservable variable. To overcome this issue, event playback is applied to simulate the model's response and a comparison between the DSE of multiple generators is made. Finally, when the faulty generator is identified, we compute energy functions for the mechanical control loop and for the excitation control loop. The sign and rate of these energies determine the source of the oscillation, allowing that the FO can be located. Simulated signals are used to show that the proposed method provides a systematic methodology for identification and location of power systems forced oscillations than even could be non-stationary signals.
Distribution network operation is becoming more challenging because of the growing integration of intermittent and volatile distributed energy resources (DERs). This motivates the development of new distribution system state estimation (DSSE) paradigms that can operate at fast timescale based on real-time data stream of asynchronous measurements enabled by modern information and communications technology. To solve the real-time DSSE with asynchronous measurements effectively and accurately, this paper formulates a weighted least squares DSSE problem and proposes an online stochastic gradient algorithm to solve it. The performance of the proposed scheme is analytically guaranteed and is numerically corroborated with realistic data on IEEE-123 bus feeder.
In this work, we propose a subsystem decomposition approach and a distributed estimation scheme for a class of implicit two-time-scale nonlinear systems. Taking the advantage of the two-time-scale separation, these processes are decomposed into smaller subsystems such as fast subsystem and slow subsystem. In the proposed method, an approach, composite solution, combines the approximate solutions obtained from both fast and slow subsystems. Based on the fast and slow subsystems, a distributed state estimation scheme is proposed to handle the implicit time-scale multiplicity. In the proposed design, an extended Kalman filter (EKF) is designed for the fast subsystem and a moving horizon estimator (MHE) is designed for the slow subsystem. There is a communication between the estimators: the slow subsystem is only required to send information to the fast subsystem one-directionally. The fast subsystem estimator does not send out any information. The estimators use different sampling of the state measurements, i.e., fast sampling of the fast state variables is considered in the fast EKF and slow sampling of the slow state variables is considered in the slow MHE. Extensive simulations based on a chemical process are performed to illustrate the effectiveness and applicability of the proposed subsystem decomposition and composite state estimation architecture.
The capability of a mobile robot to efficiently and safely perform complex missions is limited by its knowledge of the environment, namely the situation. Advanced reasoning, decision-making, and execution skills enable an intelligent agent to act autonomously in unknown environments. Situational Awareness (SA) is a fundamental capability of humans that has been deeply studied in various fields, such as psychology, military, aerospace, and education. Nevertheless, it has yet to be considered in robotics, which has focused on single compartmentalized concepts such as sensing, spatial perception, sensor fusion, state estimation, and Simultaneous Localization and Mapping (SLAM). Hence, the present research aims to connect the broad multidisciplinary existing knowledge to pave the way for a complete SA system for mobile robotics that we deem paramount for autonomy. To this aim, we define the principal components to structure a robotic SA and their area of competence. Accordingly, this paper investigates each aspect of SA, surveying the state-of-the-art robotics algorithms that cover them, and discusses their current limitations. Remarkably, essential aspects of SA are still immature since the current algorithmic development restricts their performance to only specific environments. Nevertheless, Artificial Intelligence (AI), particularly Deep Learning (DL), has brought new methods to bridge the gap that maintains these fields apart from the deployment to real-world scenarios. Furthermore, an opportunity has been discovered to interconnect the vastly fragmented space of robotic comprehension algorithms through the mechanism of Situational Graph (S-Graph), a generalization of the well-known scene graph. Therefore, we finally shape our vision for the future of robotic Situational Awareness by discussing interesting recent research directions.
An accurate and up-to-date topology is critical for situational awareness of a power grid; however, wrong switch statuses due to physical damage, communication error, or cyber-attack, can often result in topology errors. To maintain situation awareness under the possible topology errors and bad data, this paper develops ckt-GSE, a circuit-theoretic generalized state estimation method using node-breaker (NB) model. Ckt- GSE is a convex and scalable model that jointly estimates AC state variables and network topology, with robustness against different data errors. The method first constructs an equivalent circuit representation of the AC power grid by developing and aggregating linear circuit models of SCADA meters, phasor measurement units(PMUs), and switching devices. Then based on this circuit, ckt-GSE defines a constrained optimization problem using weighted least absolute value (WLAV) objective to form a robust estimator. The problem is a Linear Programming (LP) problem whose solution includes accurate AC states and a sparse vector of noise terms to identify topology errors and bad data.This paper is the first to explore a circuit-theoretic approach for an AC-network constrained GSE algorithm that is: 1) applicable to the real-world data setting, 2) convex without relaxation, scalable with our circuit-based solver; and 3) robust with the ability to identify and reject different data errors