We present a motion planning algorithm for a class of uncertain control-affine nonlinear systems which guarantees runtime safety and goal reachability when using high-dimensional sensor measurements (e.g., RGB-D images) and a learned perception module in the feedback control loop. First, given a dataset of states and observations, we train a perception system that seeks to invert a subset of the state from an observation, and estimate an upper bound on the perception error which is valid with high probability in a trusted domain near the data. Next, we use contraction theory to design a stabilizing state feedback controller and a convergent dynamic state observer which uses the learned perception system to update its state estimate. We derive a bound on the trajectory tracking error when this controller is subjected to errors in the dynamics and incorrect state estimates. Finally, we integrate this bound into a sampling-based motion planner, guiding it to return trajectories that can be safely tracked at runtime using sensor data. We demonstrate our approach in simulation on a 4D car, a 6D planar quadrotor, and a 17D manipulation task with RGB(-D) sensor measurements, demonstrating that our method safely and reliably steers the system to the goal, while baselines that fail to consider the trusted domain or state estimation errors can be unsafe.
The power system state estimation (SE) algorithm estimates the complex bus voltages based on the available set of measurements. Because phasor measurement units (PMUs) are becoming more widely employed in transmission power systems, a fast SE solver capable of exploiting PMUs' high sample rates is required. To accomplish this, we present a method for training a model based on graph neural networks (GNNs) to learn estimates from PMU voltage and current measurements, which, once it is trained, has a linear computational complexity with respect to the number of nodes in the power system. We propose an original GNN implementation over the power system's factor graph to simplify the incorporation of various types and numbers of measurements both on power system buses and branches. Furthermore, we augment the factor graph to improve the robustness of GNN predictions. Training and test examples were generated by randomly sampling sets of power system measurements and annotated with the exact solutions of linear SE with PMUs. The numerical results demonstrate that the GNN model provides an accurate approximation of the SE solutions. Additionally, errors caused by PMU malfunctions or the communication failures that make the SE problem unobservable have a local effect and do not deteriorate the results in the rest of the power system.
We consider the problem of state estimation in general state-space models using variational inference. For a generic variational family defined using the same backward decomposition as the actual joint smoothing distribution, we establish for the first time that, under mixing assumptions, the variational approximation of expectations of additive state functionals induces an error which grows at most linearly in the number of observations. This guarantee is consistent with the known upper bounds for the approximation of smoothing distributions using standard Monte Carlo methods. Moreover, we propose an amortized inference framework where a neural network shared over all times steps outputs the parameters of the variational kernels. We also study empirically parametrizations which allow analytical marginalization of the variational distributions, and therefore lead to efficient smoothing algorithms. Significant improvements are made over state-of-the art variational solutions, especially when the generative model depends on a strongly nonlinear and noninjective mixing function.
The performance of a guidance, navigation and control (GNC) system of an autonomous underwater vehicle (AUV) heavily depends on the correct tuning of its parameters. Our objective is to automatically tune these parameters with respect to arbitrary high-level control objectives within different operational scenarios. In contrast to literature, an overall tuning is performed for the entire GNC system, which is new in the context of autonomous underwater vehicles. The main challenges in solving the optimization problem are computationally expensive objective function evaluations, crashing simulations due to infeasible parametrization and the numerous tunable parameters (in our case 13). These challenges are met by using constrained Bayesian optimization with crash constraints. The method is demonstrated in simulation on a GNC system of an underactuated miniature AUV designed within the TRIPLE-nanoAUV initiative for exploration of sub-glacial lakes. We quantify the substantial reduction in energy consumption achieved by tuning the overall system. Furthermore, different parametrizations are automatically generated for different power consumption functions, robustness, and accuracy requirements. E.g. energy consumption can be reduced by ~28%, if the maximum allowed deviation from the planned path is increased by ~65%. This shows the versatile practical applicability of the optimization-based tuning approach.
Tensegrity robots, which are composed of compressive elements (rods) and flexible tensile elements (e.g., cables), have a variety of advantages, including flexibility, low weight, and resistance to mechanical impact. Nevertheless, the hybrid soft-rigid nature of these robots also complicates the ability to localize and track their state. This work aims to address what has been recognized as a grand challenge in this domain, i.e., the state estimation of tensegrity robots through a markerless, vision-based method, as well as novel, onboard sensors that can measure the length of the robot's cables. In particular, an iterative optimization process is proposed to track the 6-DoF pose of each rigid element of a tensegrity robot from an RGB-D video as well as endcap distance measurements from the cable sensors. To ensure that the pose estimates of rigid elements are physically feasible, i.e., they are not resulting in collisions between rods or with the environment, physical constraints are introduced during the optimization. Real-world experiments are performed with a 3-bar tensegrity robot, which performs locomotion gaits. Given ground truth data from a motion capture system, the proposed method achieves less than 1~cm translation error and 3 degrees rotation error, which significantly outperforms alternatives. At the same time, the approach can provide accurate pose estimation throughout the robot's motion, while motion capture often fails due to occlusions.
The fusion of camera sensor and inertial data is a leading method for ego-motion tracking in autonomous and smart devices. State estimation techniques that rely on non-linear filtering are a strong paradigm for solving the associated information fusion task. The de facto inference method in this space is the celebrated extended Kalman filter (EKF), which relies on first-order linearizations of both the dynamical and measurement model. This paper takes a critical look at the practical implications and limitations posed by the EKF, especially under faulty visual feature associations and the presence of strong confounding noise. As an alternative, we revisit the assumed density formulation of Bayesian filtering and employ a moment matching (unscented Kalman filtering) approach to both visual-inertial odometry and visual SLAM. Our results highlight important aspects in robustness both in dynamics propagation and visual measurement updates, and we show state-of-the-art results on EuRoC MAV drone data benchmark.
Variational quantum algorithms (VQAs) are expected to establish valuable applications on near-term quantum computers. However, recent works have pointed out that the performance of VQAs greatly relies on the expressibility of the ansatzes and is seriously limited by optimization issues such as barren plateaus (i.e., vanishing gradients). This work proposes the state efficient ansatz (SEA) for accurate ground state preparation with improved trainability. We show that the SEA can generate an arbitrary pure state with much fewer parameters than a universal ansatz, making it efficient for tasks like ground state estimation. Then, we prove that barren plateaus can be efficiently mitigated by the SEA and the trainability can be further improved most quadratically by flexibly adjusting the entangling capability of the SEA. Finally, we investigate a plethora of examples in ground state estimation where we obtain significant improvements in the magnitude of cost gradient and the convergence speed.
Remote state estimation, where sensors send their measurements of distributed dynamic plants to a remote estimator over shared wireless resources, is essential for mission-critical applications of Industry 4.0. Existing algorithms on dynamic radio resource allocation for remote estimation systems assumed oversimplified wireless communications models and can only work for small-scale settings. In this work, we consider remote estimation systems with practical wireless models over the orthogonal multiple-access and non-orthogonal multiple-access schemes. We derive necessary and sufficient conditions under which remote estimation systems can be stabilized. The conditions are described in terms of the transmission power budget, channel statistics, and plants' parameters. For each multiple-access scheme, we formulate a novel dynamic resource allocation problem as a decision-making problem for achieving the minimum overall long-term average estimation mean-square error. Both the estimation quality and the channel quality states are taken into account for decision making. We systematically investigated the problems under different multiple-access schemes with large discrete, hybrid discrete-and-continuous, and continuous action spaces, respectively. We propose novel action-space compression methods and develop advanced deep reinforcement learning algorithms to solve the problems. Numerical results show that our algorithms solve the resource allocation problems effectively and provide much better scalability than the literature.
In this paper we consider the problem of remote state estimation of a Gauss-Markov process, where a sensor can, at each discrete time instant, transmit on one out of M different communication channels. A key difficulty of the situation at hand is that the channel statistics are unknown. We study the case where both learning of the channel reception probabilities and state estimation is carried out simultaneously. Methods for choosing the channels based on techniques for multi-armed bandits are presented, and shown to provide stability. Furthermore, we define the performance notion of estimation regret, and derive bounds on how it scales with time for the considered algorithms.
This work considers identifying parameters characterizing a physical system's dynamic motion directly from a video whose rendering configurations are inaccessible. Existing solutions require massive training data or lack generalizability to unknown rendering configurations. We propose a novel approach that marries domain randomization and differentiable rendering gradients to address this problem. Our core idea is to train a rendering-invariant state-prediction (RISP) network that transforms image differences into state differences independent of rendering configurations, e.g., lighting, shadows, or material reflectance. To train this predictor, we formulate a new loss on rendering variances using gradients from differentiable rendering. Moreover, we present an efficient, second-order method to compute the gradients of this loss, allowing it to be integrated seamlessly into modern deep learning frameworks. We evaluate our method in rigid-body and deformable-body simulation environments using four tasks: state estimation, system identification, imitation learning, and visuomotor control. We further demonstrate the efficacy of our approach on a real-world example: inferring the state and action sequences of a quadrotor from a video of its motion sequences. Compared with existing methods, our approach achieves significantly lower reconstruction errors and has better generalizability among unknown rendering configurations.
This paper addresses two important estimation problems for linear systems, namely system identification and model-free state estimation. Our focus is on ARMAX models with unknown parameters. We first provide a reinforcement learning algorithm for system identification with guaranteed consistency. This algorithm is then used to provide a novel solution to model-free state estimation. These results are then applied to solving the model-free LQG control problem in the reinforcement learning setting.
The use of the `ship as a wave buoy analogy' (SAWB) provides a novel means to estimate sea states, where relationships are established between causal wave properties and vessel motion response information. This study focuses on a model-free machine learning approach to SAWB-based sea state estimation (SSE), using neural networks (NNs) to map vessel response spectral data to statistical wave properties for a small uninhabited surface vessel. Results showed a strong correlation between heave responses and significant wave height estimates, whilst the accuracy of mean wave period and wave heading predictions were observed to improve considerably when data from multiple vessel degrees of freedom (DOFs) was utilized. Overall, 3-DOF (heave, pitch and roll) NNs for SSE were shown to perform well when compared to existing SSE approaches that use similar simulation setups. One advantage of using small vessels for SAWB was shown as SSE accuracy was reasonable even when motion responses were low (in high-frequency, low wave height sea states). Given the information-dense statistical representation of vessel motion responses in spectral form, as well as the ability of NNs to effectively model complex relationships between variables, the designed SSE method shows promise for future adaptation to mobile SSE systems using the SAWB approach.
Object tracking is one of the fundamental problems in visual recognition tasks and has achieved significant improvements in recent years. The achievements often come with the price of enormous hardware consumption and expensive labor effort for consecutive labeling. A missing ingredient for robust tracking is achieving performance with minimal modification on network structure and semi-supervised learning intermittent labeled frames. In this paper, we ad-dress these problems in a Bayesian tracking and detection framework parameterized by neural network outputs. In our framework, the tracking and detection process is formulated in a probabilistic way as multi-objects dynamics and network detection uncertainties. With our formulation, we propose a particle filter-based approximate sampling algorithm for tracking object state estimation. Based on our particle filter inference algorithm, a semi-supervised learn-ing algorithm is utilized for learning tracking network on intermittent labeled frames by variational inference. In our experiments, we provide both mAP and probability-based detection measurements for comparison between our algorithm with non-Bayesian solutions. We also train a semi-supervised tracking network on M2Cai16-Tool-Locations Dataset and compare our results with supervised learning on fully labeled frames.
This study presents an autonomous orbit determination system based on crosslink radiometric measurements applied to a future lunar CubeSat mission to clearly highlight its advantages with respect to existing ground-based navigation strategies. This work is based on the Linked Autonomous Interplanetary Satellite Orbit Navigation (LiAISON) method which provides an autonomous navigation solution solely using satellite-to-satellite measurements, such as range and/or range-rate, to estimate absolute spacecraft states when at least one of the involved spacecraft has an orbit with a unique size, shape, and orientation. The lunar vicinity is a perfect candidate for this type of application due to the asymmetrical gravity field: the selected lunar mission, an Earth-Moon L2 (EML2) Halo orbiter, has an inter-satellite link between a lunar elliptical frozen orbiter. Simulation results show that, even in case of high-measurement errors (in the order of 100 m, 1 sigma, the navigation filter estimates the true states of spacecraft at EML2 with an error in the order of 500 m for position, and 2 mm/s for velocity, respectively and the elliptical lunar frozen orbiter states can be estimated in the order of 100 m for position and 1 cm/s for velocity, respectively. This study shows that range-only measurements provide better state estimation than range-rate-only measurements for this specific situation. Different bias handling strategies are also investigated. It has been found that even a less accurate ranging method, such as data-aided ranging, provides a sufficient orbit determination solution. This would simplify the communication system design for the selected CubeSat mission. The most observable states are found to be position states of the lunar orbiter via the observability analysis. In addition, the best tracking windows are also investigated for the selected mission scenario.
Recent advances in space technology provide an opportunity for small satellites to be launched in cislunar space. However, tracking these small satellites still depends on ground-based operations. Autonomous navigation could be a possible solution considering challenges presented by costly ground operations and limited onboard power available for small satellites. There have been various studies on autonomous navigation methods for cislunar missions. One of them, LiAISON, provides an autonomous orbit determination solution solely using inter-satellite measurements. This study aims at providing a detailed performance analysis of crosslink radiometric measurements based on autonomous orbit determination for cislunar small satellite formations considering the effects of measurement type, measurement accuracy, bias, formation geometry, and network topology. This study shows that range observations provide better state estimation accuracy than range-rate observations for the autonomous navigation system in cislunar space. Line-of-sight angle measurements derived from radiometric measurements do not improve the overall system performance. In addition, less precise crosslink measurement methods could be an option for formations in the high observable orbital configurations. It was found that measurement biases and measurements with high intervals reduce the overall system performance. In case there are more than two spacecraft in the formation, the navigation system in the mesh topology provided better overall state estimation than the centralized topology.
Satellite imaging has a central role in monitoring, detecting and estimating the intensity of key natural phenomena. One important feature of satellite images is the trade-off between spatial/spectral resolution and their revisiting time, a consequence of design and physical constraints imposed by satellite orbit among other technical limitations. In this paper, we focus on fusing multi-temporal, multi-spectral images where data acquired from different instruments with different spatial resolutions is used. We leverage the spatial relationship between images at multiple modalities to generate high-resolution image sequences at higher revisiting rates. To achieve this goal, we formulate the fusion method as a recursive state estimation problem and study its performance in filtering and smoothing contexts. The proposed strategy clearly outperforms competing methodologies, which is shown in the paper for real data acquired by the Landsat and MODIS instruments.
Fast and accurate detection of cyberattacks is a key element for a cyber-resilient power system. Recently, data-driven detectors and physics-based Moving Target Defences (MTD) have been proposed to detect false data injection (FDI) attacks on state estimation. However, the uncontrollable false positive rate of the data-driven detector and the extra cost of frequent MTD usage limit their wide applications. Few works have explored the overlap between these two areas. To fill this gap, this paper proposes blending data-driven and physics-based approaches to enhance the detection performance. To start, a physics-informed data-driven attack detection and identification algorithm is proposed. Then, an MTD protocol is triggered by the positive alarm from the data-driven detector. The MTD is formulated as a bilevel optimisation to robustly guarantee its effectiveness against the worst-case attack around the identified attack vector. Meanwhile, MTD hiddenness is also improved so that the defence cannot be detected by the attacker. To guarantee feasibility and convergence, the convex two-stage reformulation is derived through duality and linear matrix inequality. The simulation results verify that blending data and physics can achieve extremely high detection rate while simultaneously reducing the false positive rate of the data-driven detector and the extra cost of MTD. All codes are available at https://github.com/xuwkk/DDET-MTD.
Task specification is at the core of programming autonomous robots. A low-effort modality for task specification is critical for engagement of non-expert end-users and ultimate adoption of personalized robot agents. A widely studied approach to task specification is through goals, using either compact state vectors or goal images from the same robot scene. The former is hard to interpret for non-experts and necessitates detailed state estimation and scene understanding. The latter requires the generation of desired goal image, which often requires a human to complete the task, defeating the purpose of having autonomous robots. In this work, we explore alternate and more general forms of goal specification that are expected to be easier for humans to specify and use such as images obtained from the internet, hand sketches that provide a visual description of the desired task, or simple language descriptions. As a preliminary step towards this, we investigate the capabilities of large scale pre-trained models (foundation models) for zero-shot goal specification, and find promising results in a collection of simulated robot manipulation tasks and real-world datasets.
This note presents a new method for set-based joint state and parameter estimation of discrete-time systems using constrained zonotopes. This is done by extending previous set-based state estimation methods to include parameter identification in a unified framework. Unlike in interval-based methods, the existing dependencies between states and model parameters are maintained from one time step to the next, thus providing a more accurate estimation scheme. In addition, the enclosure of states and parameters is refined using measurements through generalized intersections, which are properly captured by constrained zonotopes. The advantages of the new approach are highlighted in two numerical examples.
We present a dataset of 1000 video sequences of human portraits recorded in real and uncontrolled conditions by using a handheld smartphone accompanied by an external high-quality depth camera. The collected dataset contains 200 people captured in different poses and locations and its main purpose is to bridge the gap between raw measurements obtained from a smartphone and downstream applications, such as state estimation, 3D reconstruction, view synthesis, etc. The sensors employed in data collection are the smartphone's camera and Inertial Measurement Unit (IMU), and an external Azure Kinect DK depth camera software synchronized with sub-millisecond precision to the smartphone system. During the recording, the smartphone flash is used to provide a periodic secondary source of lightning. Accurate mask of the foremost person is provided as well as its impact on the camera alignment accuracy. For evaluation purposes, we compare multiple state-of-the-art camera alignment methods by using a Motion Capture system. We provide a smartphone visual-inertial benchmark for portrait capturing, where we report results for multiple methods and motivate further use of the provided trajectories, available in the dataset, in view synthesis and 3D reconstruction tasks.