Model-free reinforcement learning algorithms have exhibited great potential in solving single-task sequential decision-making problems with high-dimensional observations and long horizons, but are known to be hard to generalize across tasks. Model-based RL, on the other hand, learns task-agnostic models of the world that naturally enables transfer across different reward functions, but struggles to scale to complex environments due to the compounding error. To get the best of both worlds, we propose a self-supervised reinforcement learning method that enables the transfer of behaviors across tasks with different rewards, while circumventing the challenges of model-based RL. In particular, we show self-supervised pre-training of model-free reinforcement learning with a number of random features as rewards allows implicit modeling of long-horizon environment dynamics. Then, planning techniques like model-predictive control using these implicit models enable fast adaptation to problems with new reward functions. Our method is self-supervised in that it can be trained on offline datasets without reward labels, but can then be quickly deployed on new tasks. We validate that our proposed method enables transfer across tasks on a variety of manipulation and locomotion domains in simulation, opening the door to generalist decision-making agents.
This paper studies a deep deterministic policy gradient (DDPG) based actor critic (AC) reinforcement learning (RL) technique to control a linear discrete-time system with a quadratic control cost while ensuring a constraint on the probability of potentially risky or undesirable events. The proposed methodology can be applied to both known and unknown system models with minor adjustments to the reward structure (negative cost). The problem is formulated by considering the average expected quadratic cost of the states and inputs over an infinite time horizon. Risky or undesirable events are represented as functions of the states at the next time step exceeding a user-defined limit. Two strategies are employed to manage the probabilistic constraint in scenarios of known and unknown system models. In the case of a known system model, the probabilistic constraint is replaced with an upper bound, such as the Chernoff bound. For unknown system models, the expected value of the indicator function of the occurrence of the risky or undesirable event is used. We have adopted a deterministic policy gradient (DPG) based AC method to derive a parameterised optimal policy. Extensive numerical simulations are performed using a second- and a fourth-order system, and the proposed method is compared with the standard risk-neutral linear quadratic regulator (LQR) and a chance-constrained model predictive control (MPC) method. The results demonstrate the effectiveness of the proposed approach in both known and unknown system model scenarios.
Most MPC (Model Predictive Control) algorithms used in industries and studied in the control academia use a two-term QP (quadratic programming), where the first term is the weighted norm of the output errors, and the second term is that of the input increments. In this work, a DMC (Dynamic Matrix Control) algorithm that uses three-term QP is studied, where the third term is the weighted norm of the output increments. In the analysis, a relationship between the three-term DMC and the two-term DMC is established; based on that, the closed-loop response curves are derived. Based on the analysis, two controller tuning procedures are developed for the three-term DMC, one for closed-loop step response and one for disturbance reduction. Finally, it will be proven that the three-term DMC can achieve a higher performance and robustness than the two-term DMC can. Simulation studies are used to demonstrate the findings and the tuning methods.
In this paper, we leverage ideas from model-based control to address the sample efficiency problem of reinforcement learning (RL) algorithms. Accelerating learning is an active field of RL highly relevant in the context of time-varying systems. Traditional transfer learning methods propose to use prior knowledge of the system behavior to devise a gradual or immediate data-driven transformation of the control policy obtained through RL. Such transformation is usually computed by estimating the performance of previous control policies based on measurements recently collected from the system. However, such retrospective measures have debatable utility with no guarantees of positive transfer in most cases. Instead, we propose a model-based transformation, such that when actions from a control policy are applied to the target system, a positive transfer is achieved. The transformation can be used as an initialization for the reinforcement learning process to converge to a new optimum. We validate the performance of our approach through four benchmark examples. We demonstrate that our approach is more sample-efficient than fine-tuning with reinforcement learning alone and achieves comparable performance to linear-quadratic-regulators and model-predictive control when an accurate linear model is known in the three cases. If an accurate model is not known, we empirically show that the proposed approach still guarantees positive transfer with jump-start improvement.
Nonlinear receding horizon model predictive control is a powerful approach to controlling nonlinear dynamical systems. However, typical approaches that use the Jacobian, adjoint, and forward-backward passes may lose fidelity and efficacy for highly nonlinear problems. Here, we develop an Ensemble Model Predictive Control (EMPC) approach wherein the forward model remains fully nonlinear, and an ensemble-represented Gaussian process performs the backward calculations to determine optimal gains for the initial time. EMPC admits black box, possible non-differentiable models, simulations are executable in parallel over long horizons, and control is uncertainty quantifying and applicable to stochastic settings. We construct the EMPC for terminal control and regulation problems and apply it to the control of a quadrotor in a simulated, identical-twin study. Results suggest that the easily implemented approach is promising and amenable to controlling autonomous robotic systems with added state/parameter estimation and parallel computing.
We propose a robust nonlinear model predictive control design with generalized zone tracking (ZMPC) in this work. The proposed ZMPC has guaranteed convergence into the target zone in the presence of bounded disturbance. The proposed approach achieves this by modifying the actual target zone such that the effect of disturbances is rejected. A control invariant set (CIS) inside the modified target zone is used as the terminal set, which ensures the closed-loop stability of the proposed controller. Detailed closed-loop stability analysis is presented. Simulation studies based on a continuous stirred tank reactor (CSTR) are performed to validate the effectiveness of the proposed ZMPC.
Inexact methods for model predictive control (MPC), such as real-time iterative schemes or time-distributed optimization, alleviate the computational burden of exact MPC by providing suboptimal solutions. While the asymptotic stability of such algorithms is well studied, their finite-time performance has not received much attention. In this work, we quantify the performance of suboptimal linear model predictive control in terms of the additional closed-loop cost incurred due to performing only a finite number of optimization iterations. Leveraging this novel analysis framework, we propose a novel suboptimal MPC algorithm with a diminishing horizon length and finite-time closed-loop performance guarantees. This analysis allows the designer to plan a limited computational power budget distribution to achieve a desired performance level. We provide numerical examples to illustrate the algorithm's transient behavior and computational complexity.
This paper studies a heterogeneous Unmanned Aerial Vehicles (UAVs) cooperative search approach suitable for complex environments. In the application, a fixed-wing UAV drops rotor UAVs to deploy the cluster rapidly. Meanwhile, the fixed-wing UAV works as a communication relay node to improve the search performance of the cluster further. The distributed model predictive control and genetic algorithms are adopted to make online intelligent decisions on UAVs search directions. On this basis, a jump grid decision method is proposed to satisfy the maneuverability constraints of UAVs, a parameter dynamic selection method is developed to make search decisions more responsive to task requirements, and a search information transmission method with low bandwidth is designed. This approach can enable UAVs to discover targets quickly, cope with various constraints and unexpected situations, and make adaptive decisions, significantly improving the robustness of search tasks in complex, dynamic, and unknown environments. The proposed approach is tested with several search scenarios, and simulation results show that the cooperative search performance of heterogeneous UAVs is significantly improved compared to homogeneous UAVs.
Genetic algorithm (GA) is typically used to solve nonlinear model predictive control's optimization problem. However, the size of the search space in which the GA searches for the optimal control inputs is crucial for its applicability to fast-response systems. This paper proposes accelerating the genetic optimization of NMPC by learning optimal search space size. The approach trains a multivariate regression model to adaptively predict the best smallest size of the search space in every control cycle. The proposed approach reduces the GA's computational time, improves the chance of convergence to better control inputs, and provides a stable and feasible solution. The proposed approach was evaluated on three nonlinear systems and compared to four other evolutionary algorithms implemented in a processor-in-the-loop fashion. The results show that the proposed approach provides a 17-45\% reduction in computational time and increases the convergence rate by 35-47\%. The source code is available on GitHub.
In continuous process plants controlled by model predictive control, model-plant mismatch (MPM), due to the aging of processes, causes degradation of control performance. We propose a concept called Model Life Extension (MLE) and its implementation to mitigate this degradation in a non-invasive manner. The purpose of MLE is to continually update (re-identify) process models by using routine operating data on the assumption that the timescale of aging is much larger than the interval of excitation of reference signals. We implemented MLE by estimating MPM via $\mathcal{L}_1$ regularized regression and by finding an optimal regularization parameter via cross-validation and showed through numerical experiments that an optimal parameter can exist and be found by cross-validation for a pilot-scale distillation column. We then constructed the updated model based on the found parameter to demonstrate the possibility of correcting static-gain mismatch and transport-delay mismatch without injecting excitation signals to process inputs.
Flexibility provided by Combined Heat and Power (CHP) units in district heating networks is an important means to cope with increasing penetration of intermittent renewable energy resources, and various methods have been proposed to exploit thermal storage tanks installed in these networks. This paper studies a novel problem motivated by an example of district heating and cooling networks in Japan, where high-temperature steam is used as the heating medium. In steam-based networks, storage tanks are usually absent, and there is a strong need to utilize thermal inertia of the pipeline network as storage. However, this type of use of a heating network directly affects the operating condition of the network, and assuring safety and supply quality at the use side is an open problem. To address this, we formulate a novel control problem to utilize CHP units in frequency regulation market while satisfying physical constraints on a steam network described by a nonlinear model capturing dynamics of heat flows and heat accumulation in the network. Furthermore, a Model Predictive Control (MPC) framework is proposed to solve this problem. By consistently combining several nonlinear control techniques, a computationally efficient MPC controller is obtained and shown to work in real-time.
The paper addresses a problem of constrained spacecraft attitude stabilization with simultaneous reaction wheel (RW) desaturation. The spacecraft has a reaction wheel array (RWA) consisting of four RWs in a pyramidal configuration. The developments exploit a spacecraft dynamics model with gravity gradient torques. The linearized dynamics are shown to be controllable at almost all RWA configurations. Configurations that result in the highest Degree of Controllability are elucidated. A strategy that combines an incremental reference governor and time-distributed model predictive control is proposed to perform constrained RW desaturation at low computational cost. Simulation results of successful RW desaturation maneuvers subject to spacecraft pointing constraints, RW zero-speed crossing avoidance and limits on control moments are reported.
The close integration of increasing operating units into an integrated energy system (IES) results in complex interconnections between these units. The strong dynamic interactions create barriers to designing a successful distributed coordinated controller to achieve synergy between all the units and unlock the potential for grid response. To address these challenges, we introduce a directed graph representation of IESs using an augmented Jacobian matrix to depict their underlying dynamics topology. By utilizing this representation, a generic subsystem decomposition method is proposed to partition the entire IES vertically based on the dynamic time scale and horizontally based on the closeness of interconnections between the operating units. Exploiting the decomposed subsystems, we develop a cooperative distributed economic model predictive control (DEMPC) with multiple global objectives that regulate the generated power at the grid's requests and satisfy the customers cooling and system economic requirements. In the DEMPC, multiple local decision-making agents cooperate sequentially and iteratively to leverage the potential across all the units for system-wide dynamic synergy. Furthermore, we discuss how subsystem decomposition impacts the design of distributed cooperation schemes for IESs and provide a control-oriented basic guideline on the optimal decomposition of complex energy systems. Extensive simulations demonstrate that the control strategies with different levels of decomposition and collaboration will lead to marked differences in the overall performance of IES. The standard control scheme based on the proposed subsystem configuration outperforms the empirical decomposition-based control benchmark by about 20%. The DEMPC architecture further improves the overall performance of the IES by about 55% compared to the benchmark.
In this paper, we show that under mild controllability assumptions a time-invariant Control Barrier Function (CBF) can be constructed based on predictions with a finite horizon. As a starting point, we require only a known subset of a control-invariant set where the latter set does not need to be explicitly known. We show that, based on ideas similar to the Hamilton-Jacobi reachability analysis, the knowledge on the subset of a control-invariant set allows us to obtain a time-invariant CBF for the time-invariant dynamics under consideration. We also provide a thorough analysis of the properties of the constructed CBF, we characterize the impact of the prediction horizon, and comment on the practical implementation. In the end, we relate our construction approach to Model Predictive Control (MPC). With a relevant application example, we demonstrate how our method is applied.
We implement a causal model predictive control (MPC) strategy to maximize power generation from a wave energy converter (WEC) system, for which the power take-off (PTO) systems have both hard stroke (i.e., displacement) limits and force ratings. The approach models the WEC dynamics in discrete-time, in a manner that exactly preserves energy-flow quantities, and assumes a stationary stochastic disturbance model for the incident wave force. The control objective is to maximize the expected power generation in stationarity, while accounting for parasitic losses in the power train. PTO stroke measurements are assumed to be available for real-time feedback, as well as the free-surface elevation of the waves at a designated location relative to the WEC, and the open-loop dynamics of the WEC are assumed to be linear and time-invariant. Mean-square stability of the MPC algorithm is proven. The methodology is illustrated in a simulation example pertaining to a heaving cylindrical buoy.
This paper addresses the trajectory-tracking problem under uncertain road-surface conditions for autonomous vehicles. We propose a stochastic nonlinear model predictive controller (SNMPC) that learns a tire--road friction model online using standard automotive-grade sensors. Learning the entire tire--road friction model in real time requires driving in the nonlinear, potentially unstable regime of the vehicle dynamics, using a prediction model that may not have fully converged. To handle this, we formulate the tire-friction model learning in a Bayesian framework, and propose two estimators that learn different aspects of the tire--road friction. The estimators output the estimate of the tire-friction model as well as the uncertainty of the estimate, which expresses the confidence in the model for different driving regimes. The SNMPC exploits the uncertainty estimate in its prediction model to take proper action when the uncertainty is large. We validate the approach in an extensive Monte-Carlo study using real vehicle parameters and in CarSim. The results when comparing to various MPC approaches indicate a substantial reduction in constraint violations, as well as a reduction in closed-loop cost. We also demonstrate the real-time feasibility in automotive-grade processors using a dSPACE MicroAutoBox-II rapid prototyping unit, showing a worst-case computation time of roughly 40ms.
Building on our recent research on neural heuristic quantization systems, results on learning quantized motions and resilience to channel dropouts are reported. We propose a general emulation problem consistent with the neuromimetic paradigm. This optimal quantization problem can be solved by model predictive control (MPC), but because the optimization step involves integer programming, the approach suffers from combinatorial complexity when the number of input channels becomes large. Even if we collect data points to train a neural network simultaneously, collection of training data and the training itself are still time-consuming. Therefore, we propose a general Deep Q Network (DQN) algorithm that can not only learn the trajectory but also exhibit the advantages of resilience to channel dropout. Furthermore, to transfer the model to other emulation problems, a mapping-based transfer learning approach can be used directly on the current model to obtain the optimal direction for the new emulation problems.
This article presents a distributed model predictive controller with time-varying partitioning based on the augmented Lagrangian alternating direction inexact Newton method (ALADIN). In particular, we address the problem of controlling the temperature of a heat transfer fluid (HTF) in a set of loops of solar parabolic collectors by adjusting its flow rate. The control problem involves a nonlinear prediction model, decoupled inequality constraints, and coupled affine constraints on the system inputs. The application of ALADIN to address such a problem is combined with a dynamic clustering-based partitioning approach that aims at reducing, with minimum performance losses, the number of variables to be coordinated. Numerical results on a 10-loop plant are presented.
The development of Adaptive Cruise Control (ACC) systems aims to enhance the safety and comfort of vehicles by automatically regulating the speed of the vehicle to ensure a safe gap from the preceding vehicle. However, conventional ACC systems are unable to adapt themselves to changing driving conditions and drivers' behavior. To address this limitation, we propose a Long Short-Term Memory (LSTM) based ACC system that can learn from past driving experiences and adapt and predict new situations in real time. The model is constructed based on the real-world highD dataset, acquired from German highways with the assistance of camera-equipped drones. We evaluated the ACC system under aggressive lane changes when the side lane preceding vehicle cut off, forcing the targeted driver to reduce speed. To this end, the proposed system was assessed on a simulated driving environment and compared with a feedforward Artificial Neural Network (ANN) model and Model Predictive Control (MPC) model. The results show that the LSTM-based system is 19.25% more accurate than the ANN model and 5.9% more accurate than the MPC model in terms of predicting future values of subject vehicle acceleration. The simulation is done in Matlab/Simulink environment.
Data-enabled predictive control (DeePC) is a recently established form of Model Predictive Control (MPC), based on behavioral systems theory. While eliminating the need to explicitly identify a model, it requires an additional regularization with a corresponding weight to function well with noisy data. The tuning of this weight is non-trivial and has a significant impact on performance. In this paper, we compare three reformulations of DeePC that either eliminate the regularization, or simplify the tuning to a trivial point. A building simulation study shows a comparable performance for all three reformulations of DeePC. However, a conventional MPC with a black-box model slightly outperforms them, while solving much faster, and yielding smoother optimal trajectories. Two of the DeePC variants also show sensitivity to an unobserved biased input noise, which is not present in the conventional MPC.