Safety is essential for reinforcement learning (RL) applied in real-world situations. Chance constraints are suitable to represent the safety requirements in stochastic systems. Previous chance-constrained RL methods usually have a low convergence rate, or only learn a conservative policy. In this paper, we propose a model-based chance constrained actor-critic (CCAC) algorithm which can efficiently learn a safe and non-conservative policy. Different from existing methods that optimize a conservative lower bound, CCAC directly solves the original chance constrained problems, where the objective function and safe probability is simultaneously optimized with adaptive weights. In order to improve the convergence rate, CCAC utilizes the gradient of dynamic model to accelerate policy optimization. The effectiveness of CCAC is demonstrated by a stochastic car-following task. Experiments indicate that compared with previous RL methods, CCAC improves the performance while guaranteeing safety, with a five times faster convergence rate. It also has 100 times higher online computation efficiency than traditional safety techniques such as stochastic model predictive control.
For linearly constrained least-squares problems that depend on a vector of parameters, this paper proposes techniques for reducing the number of involved optimization variables. After first eliminating equality constraints in a numerically robust way by QR factorization, we propose a technique based on singular value decomposition (SVD) and unsupervised learning, that we call $K$-SVD, and neural classifiers to automatically partition the set of parameter vectors in $K$ nonlinear regions in which the original problem is approximated by using a smaller set of variables. For the special case of parametric constrained least-squares problems that arise from model predictive control (MPC) formulations, we propose a novel and very efficient QR factorization method for equality constraint elimination. Together with SVD or $K$-SVD, the method provides a numerically robust alternative to standard condensing and move blocking, and to other complexity reduction methods for MPC based on basis functions. We show the good performance of the proposed techniques in numerical tests and in a linearized MPC problem of a nonlinear benchmark process.
Balancing exploration and exploitation is crucial in reinforcement learning (RL). In this paper, we study model-based posterior sampling for reinforcement learning (PSRL) in continuous state-action spaces theoretically and empirically. First, we show the first regret bound of PSRL in continuous spaces which is polynomial in the episode length to the best of our knowledge. With the assumption that reward and transition functions can be modeled by Bayesian linear regression, we develop a regret bound of $\tilde{O}(H^{3/2}d\sqrt{T})$, where $H$ is the episode length, $d$ is the dimension of the state-action space, and $T$ indicates the total time steps. This result matches the best-known regret bound of non-PSRL methods in linear MDPs. Our bound can be extended to nonlinear cases as well with feature embedding: using linear kernels on the feature representation $\phi$, the regret bound becomes $\tilde{O}(H^{3/2}d_{\phi}\sqrt{T})$, where $d_\phi$ is the dimension of the representation space. Moreover, we present MPC-PSRL, a model-based posterior sampling algorithm with model predictive control for action selection. To capture the uncertainty in models, we use Bayesian linear regression on the penultimate layer (the feature representation layer $\phi$) of neural networks. Empirical results show that our algorithm achieves the state-of-the-art sample efficiency in benchmark continuous control tasks compared to prior model-based algorithms, and matches the asymptotic performance of model-free algorithms.
Kalman Filter (KF) is widely used in various domains to perform sequential learning or variable estimation. In the context of autonomous vehicles, KF constitutes the core component of many Advanced Driver Assistance Systems (ADAS), such as Forward Collision Warning (FCW). It tracks the states (distance, velocity etc.) of relevant traffic objects based on sensor measurements. The tracking output of KF is often fed into downstream logic to produce alerts, which will then be used by human drivers to make driving decisions in near-collision scenarios. In this paper, we study adversarial attacks on KF as part of the more complex machine-human hybrid system of Forward Collision Warning. Our attack goal is to negatively affect human braking decisions by causing KF to output incorrect state estimations that lead to false or delayed alerts. We accomplish this by sequentially manipulating measure ments fed into the KF, and propose a novel Model Predictive Control (MPC) approach to compute the optimal manipulation. Via experiments conducted in a simulated driving environment, we show that the attacker is able to successfully change FCW alert signals through planned manipulation over measurements prior to the desired target time. These results demonstrate that our attack can stealthily mislead a distracted human driver and cause vehicle collisions.
This paper presents a distributed model predictive control (DMPC) algorithm for heterogeneous vehicle platoons with unidirectional topologies and a p r i o r i unknown desired set point. The vehicles (or nodes) in a platoon are dynamically decoupled but constrained by spatial geometry. Each node is assigned a local open-loop optimal control problem only relying on the information of neighboring nodes, in which the cost function is designed by penalizing on the errors between the predicted and assumed trajectories. Together with this penalization, an equalitybased terminal constraint is proposed to ensure stability, which enforces the terminal states of each node in the predictive horizon equal to the average of its neighboring states. By using the sum of local cost functions as a Lyapunov candidate, it is proved that asymptotic stability of such a DMPC can be achieved through an explicit sufficient condition on the weights of the cost functions. Simulations with passenger cars demonstrate the effectiveness of the proposed DMPC.
The combination of learning methods with Model Predictive Control (MPC) has attracted a significant amount of attention in the recent literature. The hope of this combination is to reduce the reliance of MPC schemes on accurate models, and to tap into the fast developing machine learning and reinforcement learning tools to exploit the growing amount of data available for many systems. In particular, the combination of reinforcement learning and MPC has been proposed as a viable and theoretically justified approach to introduce explainable, safe and stable policies in reinforcement learning. However, a formal theory detailing how the safety and stability of an MPC-based policy can be maintained through the parameter updates delivered by the learning tools is still lacking. This paper addresses this gap. The theory is developed for the generic Robust MPC case, and applied in simulation in the robust tube-based linear MPC case, where the theory is fairly easy to deploy in practice. The paper focuses on Reinforcement Learning as a learning tool, but it applies to any learning method that updates the MPC parameters online.
Model-Predictive Control (MPC) is a powerful tool for controlling complex, real-world systems that uses a model to make predictions about future behavior. For each state encountered, MPC solves an online optimization problem to choose a control action that will minimize future cost. This is a surprisingly effective strategy, but real-time performance requirements warrant the use of simple models. If the model is not sufficiently accurate, then the resulting controller can be biased, limiting performance. We present a framework for improving on MPC with model-free reinforcement learning (RL). The key insight is to view MPC as constructing a series of local Q-function approximations. We show that by using a parameter $\lambda$, similar to the trace decay parameter in TD($\lambda$), we can systematically trade-off learned value estimates against the local Q-function approximations. We present a theoretical analysis that shows how error from inaccurate models in MPC and value function estimation in RL can be balanced. We further propose an algorithm that changes $\lambda$ over time to reduce the dependence on MPC as our estimates of the value function improve, and test the efficacy our approach on challenging high-dimensional manipulation tasks with biased models in simulation. We demonstrate that our approach can obtain performance comparable with MPC with access to true dynamics even under severe model bias and is more sample efficient as compared to model-free RL.
As wind power penetration increases, the wind farms are required by newly released grid codes to provide frequency regulation service. The most critical challenge is how to formulate the dynamic model of wind farm for dynamic control, since it is essentially is nonlinear and there are huge amount of parameters to be maintained frequently. This paper proposes a data-driven model predictive control (data-driven MPC) method to make wind farms participate primary frequency regulation. In this method,a specialized dynamic mode decomposition (SDMD) algorithm is developed, which can linearly approximate the dynamics of wind farm from measurements based on Koopman operator theory.Compared with the existing extended dynamic mode decomposition (EDMD) method,this tailored SDMD has two advantages: 1) fully capturing the nonlinear transients of wind turbine dynamics with good accuracy under a wide range of working conditions; 2) much less computational burden with model dimensionality reduction. Based on the recursively updated linear dynamic model, a model predictive control solution is implemented. The simulation results show this model-free solution can dynamically optimize wind turbine generators' active power to track the frequency response requirement from system operator and minimize the rotor speed distortion.
We propose model-plant mismatch learning offset-free model predictive control (MPC), which learns and applies the intrinsic model-plant mismatch, to effectively exploit the advantages of model-based and data-driven control strategies and overcome the limitations of each approach. In this study, the model-plant mismatch map on steady-state manifold in the controlled variable space is approximated via a general regression neural network from the steady-state data for each setpoint. Though the learned model-plant mismatch map can provide the information at the equilibrium point (i.e., setpoint), it cannot provide model-plant mismatch information during the transient state. Moreover, the intrinsic model-plant mismatch can vary due to system characteristics changes during operation. Therefore, we additionally apply a supplementary disturbance variable which is updated from the disturbance estimator based on the nominal offset-free MPC scheme. Then, the combined disturbance signal is applied to the target problem and finite-horizon optimal control problem of offset-free MPC to improve the prediction accuracy and closed-loop performance of the controller. By this, we can exploit both the learned model-plant mismatch information and the stabilizing property of the nominal disturbance estimator approach. The closed-loop simulation results demonstrate that the developed scheme can properly learn the intrinsic model-plant mismatch and efficiently improve the model-plant mismatch compensating performance in offset-free MPC. Moreover, we examine the robust asymptotic stability of the developed offset-free MPC scheme, which is known to be difficult to analyze in nominal offset-free MPC, by exploiting the learned model-plant mismatch information.
A novel nonlinear model predictive control approach for state signal shaping is proposed. The control strategy introduces a residual shape cost kernel based on the dynamics of circular limit cycles from a supercritical Neimark-Sacker bifurcation normal form. This allows the controller to impose a fundamental harmonic state signal shape with a specific frequency and amplitude. An application example for harmonic compensation in distribution grids integrated with renewable energies is presented. The controller is tasked with the calculation of the reference current for an active power filter used for load compensation. The results achieved are successful, reducing the harmonic distortion to satisfactory levels while ensuring the correct frequency and amplitude.
We describe the architecture and algorithms of the Adaptive Charging Network (ACN), which was first deployed on the Caltech campus in early 2016 and is currently operating at over 100 other sites in the United States. The architecture enables real-time monitoring and control and supports electric vehicle (EV) charging at scale. The ACN adopts a flexible Adaptive Scheduling Algorithm based on convex optimization and model predictive control and allows for significant over-subscription of electrical infrastructure. We describe some of the practical challenges in real-world charging systems, including unbalanced three-phase infrastructure, non-ideal battery charging behavior, and quantized control signals. We demonstrate how the Adaptive Scheduling Algorithm handles these challenges, and compare its performance against baseline algorithms from the deadline scheduling literature using real workloads recorded from the Caltech ACN and accurate system models. We find that in these realistic settings, our scheduling algorithm can improve operator profit by 3.4 times over uncontrolled charging and consistently outperforms baseline algorithms when delivering energy in highly congested systems.
State-of-the-art Model Predictive Control (MPC) applications for building heating adopt either a deterministic controller together with a nonlinear model or a linearized model with a stochastic MPC controller. However, deterministic MPC only considers one single realization of the disturbances and its performance strongly depends on the quality of the forecast of the disturbances, which can lead to low performance. In fact, inadequate building energy management can lead to high energy costs and CO$_2$ emissions. On the other hand, a linearized model can fail to capture some dynamics and behavior of the building under control. In this article, we combine a stochastic scenario-based MPC (SBMPC) controller together with a nonlinear Modelica model that is able to provide a richer building description and to capture the dynamics of the building more accurately than linear models. The adopted SBMPC controller considers multiple realizations of the external disturbances obtained through a statistically accurate model, so as to consider different possible disturbance evolutions and to robustify the control action. To this purpose, we present a scenario generation method for building temperature control that can be applied to several exogenous perturbations, e.g.\ solar irradiance, outside temperature, and that satisfies several important stastistical properties, in contrast with simpler and less accurate methods adopted in the literature. We show the benefits of our proposed approach through several simulations in which we compare our method against the standard ones from the literature, for several combinations of a trade-off parameter between comfort and energy cost. We show how our SBMPC controller approach outperforms the standard controllers available in the literature.
In this paper, we propose a suboptimal moving horizon estimator for nonlinear systems. For the stability analysis we transfer the "feasibility-implies-stability/robustness" paradigm from model predictive control to the context of moving horizon estimation in the following sense: Using a suitably defined, feasible candidate solution based on the trajectory of an auxiliary observer, robust stability of the proposed suboptimal estimator is inherited independently of the horizon length and even if no optimization is performed.
The modular open-source framework GRAMPC-D for model predictive control of distributed systems is presented in this paper. The modular concept allows to solve optimal control problems (OCP) in a centralized and distributed fashion using the same problem description. It is tailored to computational efficiency with the focus on embedded hardware. The distributed solution is based on the Alternating Direction Method of Multipliers (ADMM) and uses the concept of neighbor approximation to enhance convergence speed. The presented framework can be accessed through Cpp and Python and also supports plug-and-play and data exchange between agents over a network.
Optimizing pump operations is a challenging task for real-time management of water distribution systems (WDSs). With suitable pump scheduling, pumping costs can be significantly reduced. In this research, a novel economic model predictive control (EMPC) framework for real-time management of WDSs is proposed. Optimal pump operations are selected based on predicted system behavior over a receding time horizon with the aim to minimize the total pumping energy cost. Time-varying electricity tariffs are considered while all the required water demands are satisfied. The novelty of this framework is to choose the number of pumps to operate in each pump station as decision variables in order to optimize the total pumping energy costs. By using integer programming, the proposed EMPC is applied to a benchmark case study, the Richmond Pruned network. The simulation with an EPANET hydraulic simulator is implemented. Moreover, a comparison of the results obtained using the proposed EMPC with those obtained using trigger-level control demonstrates significant economic benefits of the proposed EMPC.
In urban drainage systems (UDS), a proven method for reducing the combined sewer overflow (CSO) pollution is real-time control (RTC) based on model predictive control (MPC). MPC methodologies for RTC of UDSs in the literature rely on the computation of the optimal control strategies based on deterministic rain forecast. However, in reality, uncertainties exist in rainfall forecasts which affect severely accuracy of computing the optimal control strategies. Under this context, this work aims to focus on the uncertainty associated with the rainfall forecasting and its effects. One option is to use stochastic information about the rain events in the controller; in the case of using MPC methods, the class called stochastic MPC is available, including several approaches such as the chance-constrained MPC method. In this study, we apply stochastic MPC to the UDS using the chance-constrained method. Moreover, we also compare the operational behavior of both the classical MPC with perfect forecast and the chance-constrained MPC based on different stochastic scenarios of the rain forecast. The application and comparison have been based on simulations using a SWMM model of the Astlingen urban drainage benchmark network.
In this paper, we develop a grid-interactive multi-zone building controller based on a deep reinforcement learning (RL) approach. The controller is designed to facilitate building operation during normal conditions and demand response events, while ensuring occupants comfort and energy efficiency. We leverage a continuous action space RL formulation, and devise a two-stage global-local RL training framework. In the first stage, a global fast policy search is performed using a gradient-free RL algorithm. In the second stage, a local fine-tuning is conducted using a policy gradient method. In contrast to the state-of-the-art model predictive control (MPC) approach, the proposed RL controller does not require complex computation during real-time operation and can adapt to non-linear building models. We illustrate the controller performance numerically using a five-zone commercial building.
This paper designs a model predictive control (MPC) law for constrained linear systems with stochastic additive disturbances and noisy measurements, minimising a discounted cost subject to a discounted expectation constraint. It is assumed that sensor data is lost with a known probability. Taking into account the data losses modelled by a Bernoulli process, we parameterise the predicted control policy as an affine function of future observations and obtain a convex linear-quadratic optimal control problem. Constraint satisfaction and a discounted cost bound are ensured without imposing bounds on the distributions of the disturbance and noise inputs. In addition, the average long-run undiscounted closed loop cost is shown to be finite if the discount factor takes appropriate values. We analyse robustness of the proposed control law with respect to possible uncertainties in the arrival probability of sensor data and we bound the impact of these uncertainties on constraint satisfaction and the discounted cost. Numerical simulations are provided to illustrate these results.
Model predictive control (MPC) is a widely used modern control technique with numerous successful application in diverse areas. Much of this success is due to the ability of MPC to enforce state and control constraints, which are crucial in many applications of control. In order to avoid the need for an observer, output-feedback model predictive control with online identification (OFMPCOI) uses the block observable canonical form whose state consists of past values of the control inputs and measured outputs. Online identification is performed using recursive least squares (RLS) with variable-rate forgetting. The article describes the algorithmic details of OFMPCOI and numerically investigates its performance through a collection of numerical examples that highlight various control challenges, such as model order uncertainty, sensor noise, prediction horizon, stabilization, magnitude and move-size saturation, and stabilization. The numerical examples are used to probe the performance of OFMPCOI in terms of persistency, consistency, and exigency. Since OFMPCOI does not employ a separate control perturbation to enhance persistency, the focus is on self-generated persistency during transient operation. For closed-loop identification using RLS, sensor noise gives rise to bias in the identified model, and the goal is to determine the effect of the lack of consistency. Finally, the numerical examples reveal exigency, which is the extent to which the online identification emphasizes model characteristics that are most relevant to meeting performance objectives.
This paper experimentally demonstrates the effectiveness of an anticipative car-following algorithm in reducing energy use of gasoline engine and electric Connected and Automated Vehicles (CAV), without sacrificing safety and traffic flow. We propose a Vehicle-in-the-Loop (VIL) testing environment in which experimental CAVs driven on a track interact with surrounding virtual traffic in real-time. We explore the energy savings when following city and highway drive cycles, as well as in emergent highway traffic created from microsimulations. Model predictive control handles high level velocity planning and benefits from communicated intentions of a preceding CAV or estimated probable motion of a preceding human driven vehicle. A combination of classical feedback control and data-driven nonlinear feedforward control of pedals achieve acceleration tracking at the low level. The controllers are implemented in ROS and energy is measured via calibrated OBD-II readings. We report up to 30% improved energy economy compared to realistically calibrated human driver car-following without sacrificing following headway.