Building upon prior research that highlighted the need for standardizing environments for building control research, and inspired by recently introduced challenges for real life reinforcement learning control, here we propose a non-exhaustive set of nine real world challenges for reinforcement learning control in grid-interactive buildings. We argue that research in this area should be expressed in this framework in addition to providing a standardized environment for repeatability. Advanced controllers such as model predictive control and reinforcement learning (RL) control have both advantages and disadvantages that prevent them from being implemented in real world problems. Comparisons between the two are rare, and often biased. By focusing on the challenges, we can investigate the performance of the controllers under a variety of situations and generate a fair comparison. As a demonstration, we implement the offline learning challenge in CityLearn and study the impact of different levels of domain knowledge and complexity of RL algorithms. We show that the sequence of operations utilized in a rule based controller (RBC) used for offline training affects the performance of the RL agents when evaluated on a set of four energy flexibility metrics. Longer offline learning from an optimized RBC leads to improved performance in the long run. RL agents that learn from a simplified RBC risk poorer performance as the offline learning period increases. We also observe no impact on performance from information sharing amongst agents. We call for a more interdisciplinary effort of the research community to address the real world challenges, and unlock the potential of grid-interactive building
The typical agricultural irrigation scheduler provides information on how much to irrigate and when to irrigate. The accurate and effective scheduler decision for a large agricultural field is still an open research problem. In this work, we address the high dimensionality of the agricultural field and propose a systematic approach to provide optimum irrigation amount and irrigation time for three-dimensional agro-hydrological systems. The water dynamics of the agro-hydrological system are represented using a cylindrical three-dimensional Richards Equation. We introduce a structure-preserving model reduction technique to decrease the dimension of the system model. Using the reduced model, the optimization-based closed-loop scheduler is designed in model predictive control (MPC) environment. The closed-loop approach can handle weather disturbances and provide improved yield and water conservation. The primary objective of the proposed scheduler is to ensure maximum yield, minimum water consumption and maximize the time between the two irrigation events, which results in less electricity usage. The proposed approach is applied to three different scenarios to show the effectiveness and superiority of the proposed framework.
The development of well-devised irrigation scheduling methods is desirable from the perspectives of plant quality and water conservation. In this article, a model predictive control (MPC) with discrete actuators is developed for irrigation scheduling, where a long short-term memory (LSTM) model of the soil-water-atmosphere system is used to evaluate the objective of ensuring optimal water uptake in crops while minimizing total water consumption and irrigation costs. A heuristic method involving a sigmoid function is used in this framework to enhance the computational efficiency of the scheduler. The scheduling scheme is applied to homogeneous and spatially variable fields and the results indicate that the LSTM-based MPC with discrete actuators is able to prescribe optimal or near-optimal irrigation schedules that are typical of irrigation practice.
In this paper, we present a robust distributed model predictive control (DMPC) scheme for dynamically decoupled nonlinear systems which are subject to state constraints, coupled state constraints and input constraints. In the proposed control scheme, all subsystems solve their local optimization problem in parallel and neighbor-to-neighbor communication suffices. The approach relies on consistency constraints which define a neighborhood around each subsystem's reference trajectory where the state of the subsystem is guaranteed to stay in. Reference trajectories and consistency constraints are known to neighboring subsystems. Contrary to other relevant approaches, the reference trajectories are improved consecutively. The presented approach allows the formulation of convex optimization problems for systems with linear dynamics even in the presence of non-convex state constraints. Additionally, we employ tubes in order to ensure the controller's robustness against bounded uncertainties. In the end, we briefly comment on an iterative extension of the DMPC scheme. The effectiveness of the proposed DMPC scheme and its iterative extension are demonstrated with simulations.
Energy efficiency and safety are two critical objectives for marine vehicles operating in environments with obstacles, and they generally conflict with each other. In this paper, we propose a novel online motion planning method of marine vehicles which can make trade-offs between the two design objectives based on the framework of economic model predictive control (EMPC). Firstly, the feasible trajectory with the most safety margin is designed and utilized as tracking reference. Secondly, the EMPC-based receding horizon motion planning algorithm is designed, in which the practical consumed energy and safety measure (i.e., the distance between the planning trajectory and the reference) are considered. Experimental results verify the effectiveness and feasibility of the proposed method.
This article presents a sparse, low-memory footprint optimization algorithm for the implementation of the model predictive control (MPC) for tracking formulation in embedded systems. This MPC formulation has several advantages over standard MPC formulations, such as an increased domain of attraction and guaranteed recursive feasibility even in the event of a sudden reference change. However, this comes at the expense of the addition of a small amount of decision variables to the MPC's optimization problem that complicates the structure of its matrices. We propose a sparse optimization algorithm, based on an extension of the alternating direction method of multipliers, that exploits the structure of this particular MPC formulation. We describe the controller formulation and detail how its structure is exploited by means of the aforementioned optimization algorithm. We show closed-loop simulations comparing the proposed solver against other solvers and approaches from the literature.
A powerful result from behavioral systems theory known as the fundamental lemma allows for predictive control akin to Model Predictive Control (MPC) for linear time invariant (LTI) systems with unknown dynamics purely from data. While most of data-driven predictive control literature focuses on robustness with respect to measurement noise, only few works consider exploiting probabilistic information of disturbances for performance-oriented control as in stochastic MPC. In this work, we propose a novel data-driven stochastic predictive control scheme for chance-constrained LTI systems subject to measurement noise and additive stochastic disturbances. In order to render the otherwise stochastic and intractable optimal control problem deterministic, our approach leverages ideas from tube-based MPC by decomposing the state into a deterministic nominal state driven by inputs and a stochastic error state affected by disturbances. By tightening constraints probabilistically with respect to the additive disturbance and robustly with respect to the measurement noise, satisfaction of original chance-constraints is guaranteed. The resulting data-driven receding horizon optimal control problem is lightweight and recursively feasible, and renders the closed loop input-to-state stable with respect to both additive disturbances and measurement noise. We demonstrate the effectiveness of the proposed approach in a simulation example.
Encrypted control seeks confidential controller evaluation in cloud-based or networked systems. Many existing approaches build on homomorphic encryption (HE) that allow simple mathematical operations to be carried out on encrypted data. Unfortunately, HE is computationally demanding and many control laws (in particular non-polynomial ones) cannot be efficiently implemented with this technology. We show in this paper that secure two-party computation using garbled circuits provides a powerful alternative to HE for encrypted control. More precisely, we present a novel scheme that allows to efficiently implement (non-polynomial) max-out neural networks with one hidden layer in a secure fashion. These networks are of special interest for control since they allow, in principle, to exactly describe piecewise affine control laws resulting from, e.g., linear model predictive control (MPC). However, exact fits require high-dimensional preactivations of the neurons. Fortunately, we illustrate that even low-dimensional learning-based approximations are sufficiently accurate for linear MPC. In addition, these approximations can be securely evaluated using garbled circuit in less than 100~ms for our numerical example. Hence, our approach opens new opportunities for applying encrypted control.
A common approach in robotics is to learn tasks by generalizing from special cases given by a so-called demonstrator. In this paper, we apply this paradigm and present an algorithm that uses a demonstrator (typically given by a trajectory optimizer) to automatically synthesize feedback controllers for steering a system described by ordinary differential equations into a goal set. The resulting feedback control law switches between the demonstrations that it uses as reference trajectories. In comparison to the direct use of trajectory optimization as a control law, for example, in the form of model predictive control, this allows for a much simpler and more efficient implementation of the controller. The synthesis algorithm comes with rigorous convergence and optimality results, and computational experiments confirm its efficiency.
Policy Search and Model Predictive Control~(MPC) are two different paradigms for robot control: policy search has the strength of automatically learning complex policies using experienced data, while MPC can offer optimal control performance using models and trajectory optimization. An open research question is how to leverage and combine the advantages of both approaches. In this work, we provide an answer by using policy search for automatically choosing high-level decision variables for MPC, which leads to a novel policy-search-for-model-predictive-control framework. Specifically, we formulate the MPC as a parameterized controller, where the hard-to-optimize decision variables are represented as high-level policies. Such a formulation allows optimizing policies in a self-supervised fashion. We validate this framework by focusing on a challenging problem in agile drone flight: flying a quadrotor through fast-moving gates. Experiments show that our controller achieves robust and real-time control performance in both simulation and the real world. The proposed framework offers a new perspective for merging learning and control.
The wide adoption of wireless devices in the Internet of Things requires controllers that are able to operate with limited resources, such as battery life. Operating these devices robustly in an uncertain environment, while managing available resources, increases the difficultly of controller design. This paper proposes a robust self-triggered model predictive control approach to optimize a control objective while managing resource consumption. In particular, a novel zero-order-hold aperiodic discrete-time feedback control law is developed to ensure robust constraint satisfaction for continuous-time linear systems.
We study the problem of online learning in predictive control of an unknown linear dynamical system with time varying cost functions which are unknown apriori. Specifically, we study the online learning problem where the control algorithm does not know the true system model and has only access to a fixed-length (that does not grow with the control horizon) preview of the future cost functions. The goal of the online algorithm is to minimize the dynamic regret, defined as the difference between the cumulative cost incurred by the algorithm and that of the best sequence of actions in hindsight. We propose two different online Model Predictive Control (MPC) algorithms to address this problem, namely Certainty Equivalence MPC (CE-MPC) algorithm and Optimistic MPC (O-MPC) algorithm. We show that under the standard stability assumption for the model estimate, the CE-MPC algorithm achieves $\mathcal{O}(T^{2/3})$ dynamic regret. We then extend this result to the setting where the stability assumption holds only for the true system model by proposing the O-MPC algorithm. We show that the O-MPC algorithm also achieves $\mathcal{O}(T^{2/3})$ dynamic regret, at the cost of some additional computation. We also present numerical studies to demonstrate the performance of our algorithm.
This work deals with the problem of simultaneous regulation and model parameter estimation in adaptive model predictive control. We propose an adaptive model predictive control and conditions which guarantee a persistently exciting closed loop sequence by only looking forward in time into the receding prediction horizon. Earlier works needed to look backwards and preserve prior regressor data. Instead, we present a procedure for the offline generation of a persistently exciting reference trajectory perturbing the equilibrium. With the new approach we demonstrate exponential convergence of nonlinear systems under the influence of the adaptive model predictive control combined with a recursive least squares identifier with forgetting factor despite bounded noise. The results are, at this stage, local in state and parameter-estimate space.
This paper presents a method to identify an uncertain linear time-invariant (LTI) prediction model for tube-based Robust Model Predictive Control (RMPC). The uncertain model is determined from a given state-input dataset by formulating and solving a Semidefinite Programming problem (SDP), that also determines a static linear feedback gain and corresponding invariant sets satisfying the inclusions required to guarantee recursive feasibility and stability of the RMPC scheme, while minimizing an identification criterion. As demonstrated through an example, the proposed concurrent approach provides less conservative invariant sets than a sequential approach.
Selective laser melting is a promising additive manufacturing technology enabling the fabrication of highly customizable products. A major challenge in selective laser melting is ensuring the quality of produced parts, which is influenced greatly by the thermal history of printed layers. We propose a Batch-Model Predictive Control technique based on the combination of model predictive control and iterative learning control. This approach succeeds in rejecting both repetitive and non-repetitive disturbances and thus achieves improved tracking performance and process quality. In a simulation study, the selective laser melting dynamics is approximated with a reduced-order control-oriented linear model to ensure reasonable computational complexity. The proposed approach provides convergence to the desired temperature field profile despite model uncertainty and disturbances.
Model predictive control is a control approach that minimizes a stage cost over a predicted system trajectory based on a model of the system and is capable of handling state and input constraints. For uncertain models, robust or adaptive methods can be used. Because the system model is used to calculate the control law, the closed-loop behavior of the system and thus its performance, measured by the sum of the stage costs, are related to the model used. If it is adapted online, a performance bound is difficult to obtain and thus the impact of model adaptation is mostly unknown. This work provides a (worst-case) performance bound for a linear adaptive predictive control scheme with a specific model parameter estimation. The proposed bound is expressed in terms of quantities such as the initial system parameter error and the constraint set, among others and can be calculated a priori. The results are discussed in a numerical example.
Coasting has been widely used in the eco-driving guidelines to reduce fuel consumption by profiting from kinetic energy. However, the comprehensive comparison between different coasting strategies and online performance of the eco-coasting strategy using road grade preview are still unclear because of the oversimplification and the integer variable in the optimal control problems. Herein, two different coasting strategies (fuel cut-off and engine start/stop) are proposed to reveal the potential benefit of eco-coasting using the road grade preview. Engine drag torque and energy cost used for engine restart are considered in the modeling to give a fair evaluation of the offline and online performance. The offline performance of these two coasting methods is evaluated through dynamic programming (DP) under various driving scenarios with different slope profiles. Offline simulation shows that the engine start/stop method outperforms the fuel cut-off method in terms of fuel consumption and travel time by getting rid of the engine drag torque. Then, online performance of these two coasting methods is evaluated using Mixed Integer Model Predictive Control (MIMPC). A novel operational constraint on the minimum off steps is added in the MIMPC formulation to avoid frequent switch of the integer variables which represent the fuel cut-off and the engine start/stop mechanism. Simulation results show that, for both fuel cut-off and engine start/stop coasting methods, the MPC controller reduces fuel consumption to a level comparable to DP without sacrificing the travel time.
A common problem affecting neural network (NN) approximations of model predictive control (MPC) policies is the lack of analytical tools to assess the stability of the closed-loop system under the action of the NN-based controller. We present a general procedure to quantify the performance of such a controller, or to design minimum complexity NNs with rectified linear units (ReLUs) that preserve the desirable properties of a given MPC scheme. By quantifying the approximation error between NN-based and MPC-based state-to-input mappings, we first establish suitable conditions involving two key quantities, the worst-case error and the Lipschitz constant, guaranteeing the stability of the closed-loop system. We then develop an offline, mixed-integer optimization-based method to compute those quantities exactly. Together these techniques provide conditions sufficient to certify the stability and performance of a ReLU-based approximation of an MPC control law.
Multi-agent formation as well as obstacle avoidance is one of the most actively studied topics in the field of multi-agent systems. Although some classic controllers like model predictive control (MPC) and fuzzy control achieve a certain measure of success, most of them require precise global information which is not accessible in harsh environments. On the other hand, some reinforcement learning (RL) based approaches adopt the leader-follower structure to organize different agents' behaviors, which sacrifices the collaboration between agents thus suffering from bottlenecks in maneuverability and robustness. In this paper, we propose a distributed formation and obstacle avoidance method based on multi-agent reinforcement learning (MARL). Agents in our system only utilize local and relative information to make decisions and control themselves distributively. Agent in the multi-agent system will reorganize themselves into a new topology quickly in case that any of them is disconnected. Our method achieves better performance regarding formation error, formation convergence rate and on-par success rate of obstacle avoidance compared with baselines (both classic control methods and another RL-based method). The feasibility of our method is verified by both simulation and hardware implementation with Ackermann-steering vehicles.
We propose a robust model predictive control (MPC) method for discrete-time linear time-invariant systems with norm-bounded additive disturbances and model uncertainty. In our method, at each time step we solve a finite time robust optimal control problem (OCP) which jointly searches over robust linear state feedback controllers and bounds the deviation of the system states from the nominal predicted trajectory. By leveraging the System Level Synthesis (SLS) framework, the proposed robust OCP is formulated as a convex quadratic program in the space of closed-loop system responses. When an adaptive horizon strategy is used, we prove the recursive feasibility of the proposed MPC controller and input-to-state stability of the origin for the closed-loop system. We demonstrate through numerical examples that the proposed method considerably reduces conservatism when compared with existing SLS-based and tube-based robust MPC methods, while also enjoying low computational complexity.