Suboptimal methods in optimal control arise due to a limited computational budget, unknown system dynamics, or a short prediction window among other reasons. Although these methods are ubiquitous, their transient performance remains relatively unstudied. We consider the control of discrete-time, nonlinear time-varying dynamical systems and establish sufficient conditions to analyze the finite-time closed-loop performance of such methods in terms of the additional cost incurred due to suboptimality. Finite-time guarantees allow the control design to distribute a limited computational budget over a time horizon and estimate the on-the-go loss in performance due to suboptimality. We study exponential incremental input-to-state stabilizing policies and show that for nonlinear systems, under some mild conditions, this property is directly implied by exponential stability without further assumptions on global smoothness. The analysis is showcased on a suboptimal model predictive control use case.
Path signatures have been proposed as a powerful representation of paths that efficiently captures the path's analytic and geometric characteristics, having useful algebraic properties including fast concatenation of paths through tensor products. Signatures have recently been widely adopted in machine learning problems for time series analysis. In this work we establish connections between value functions typically used in optimal control and intriguing properties of path signatures. These connections motivate our novel control framework with signature transforms that efficiently generalizes the Bellman equation to the space of trajectories. We analyze the properties and advantages of the framework, termed signature control. In particular, we demonstrate that (i) it can naturally deal with varying/adaptive time steps; (ii) it propagates higher-level information more efficiently than value function updates; (iii) it is robust to dynamical system misspecification over long rollouts. As a specific case of our framework, we devise a model predictive control method for path tracking. This method generalizes integral control, being suitable for problems with unknown disturbances. The proposed algorithms are tested in simulation, with differentiable physics models including typical control and robotics tasks such as point-mass, curve following for an ant model, and a robotic manipulator.
In this paper, we introduce a new class of parameterized controllers, drawing inspiration from Model Predictive Control (MPC). The controller resembles a Quadratic Programming (QP) solver of a linear MPC problem, with the parameters of the controller being trained via Deep Reinforcement Learning (DRL) rather than derived from system models. This approach addresses the limitations of common controllers with Multi-Layer Perceptron (MLP) or other general neural network architecture used in DRL, in terms of verifiability and performance guarantees, and the learned controllers possess verifiable properties like persistent feasibility and asymptotic stability akin to MPC. On the other hand, numerical examples illustrate that the proposed controller empirically matches MPC and MLP controllers in terms of control performance and has superior robustness against modeling uncertainty and noises. Furthermore, the proposed controller is significantly more computationally efficient compared to MPC and requires fewer parameters to learn than MLP controllers. Real-world experiments on vehicle drift maneuvering task demonstrate the potential of these controllers for robotics and other demanding control tasks.
The objective of this work is (i) to develop an encrypted cloud-based HVAC control framework to ensure the privacy of occupancy information, (ii) to reduce the communication and computation costs of encrypted HVAC control,(iii) to reduce the leakage of private information via the triggering time instances. Occupancy of a building is sensitive and private information that can be accurately inferred by cloud-based HVAC controllers. To ensure the privacy of the privacy information, in our framework, the measurements of an HVAC system are encrypted by a fully homomorphic encryption prior to communication with the cloud controller. We first develop an encrypted algorithm that allows the cloud controller to regulate the indoor temperature and CO_2 of a building. We next develop an event-triggered control policy to reduce the communication and computation costs of the encrypted HVAC control. We cast the optimal design of the event-triggered policy as an optimal control problem. Using Bellman's optimality principle, we study the structural properties of the optimal event-triggered policy and show the necessary information for optimal triggering policy. We also show that the optimal design of the event-triggered policy can be transformed into a Markov decision process by introducing new states. As the triggering time instances are not encrypted, there is a risk that the cloud may use them to deduce sensitive information. To mitigate this risk, we introduce two randomized triggering strategies. We finally study the performance of the developed encrypted HVAC control framework using the TRNSYS simulator. Our numerical results show that the proposed framework not only ensures efficient control of the indoor temperature and CO$_2$ but also reduces the computation and communication costs of encrypted HVAC control by at least 60%.
This paper presents a novel approach to multi-agent reinforcement learning (RL) for linear systems with convex polytopic constraints. Existing work on RL has demonstrated the use of model predictive control (MPC) as a function approximator for the policy and value functions. The current paper is the first work to extend this idea to the multi-agent setting. We propose the use of a distributed MPC scheme as a function approximator, with a structure allowing for distributed learning and deployment. We then show that Q-learning updates can be performed distributively without introducing nonstationarity, by reconstructing a centralized learning update. The effectiveness of the approach is demonstrated on two numerical examples.
Data-driven learning-based control methods such as reinforcement learning (RL) have become increasingly popular with recent proliferation of the machine learning paradigm. These methods address the parameter sensitiveness and unmodeled dynamics in model-based controllers, such as finite control-set model predictive control. RL agents are typically utilized in simulation environments, where they are allowed to explore multiple "unsafe" actions during the learning process. However, this type of learning is not applicable to online self-learning of controllers in physical power converters, because unsafe actions would damage them. To address this, this letter proposes a safe online RL-based control framework to autonomously find the optimal switching strategy for the power converters, while ensuring system safety during the entire self-learning process. The proposed safe online RL-based control is validated in a practical testbed on a two-level voltage source converter system, and the results confirm the effectiveness of the proposed method.
Residential electric water heaters have significant load shifting capabilities due to their thermal heat capacity and large energy consumption. Model predictive control (MPC) has been shown to be an effective control strategy to enable water heater load shifting in home energy management systems. In this work, we analyze how modeling tank stratification in an MPC formulation impacts control performance for stratified electric water heaters under time-of-use (TOU) rates. Specifically, we propose an MPC formulation based on a three-node thermal model that captures tank stratification, and compare it to a one-node formulation that does not capture stratification and a standard thermostatic controller. These strategies are compared through both real-time laboratory testing and simulation-based evaluation for different water use patterns. Laboratory experiments show cost reductions of 12.3-23.2% for the one-node MPC and 31.2-42.5% for the three-node MPC relative to the thermostatic controller. The performance of the one-node MPC is limited by significant plant-model mismatch, while the three-node formulation better approximates real-world dynamics and results in much more effective cost reduction and load shifting. A simple analysis of how each strategy performs under water use forecast errors is also provided.
This paper presents a novel control strategy for medium voltage DC (MVDC) naval shipboard microgrids (MGs), employing a nonlinear model predictive controller (NMPC) enhanced with stabilizing features and an intricate droop control architecture. This combination quickly regulates the output voltage and adeptly allocates supercapacitors for pulsed power loads (PPLs), while the battery energy storage system (BESS) and auxiliary generators handle the steady state loads. A key feature of this study is the formulation of terminal cost and constraints, providing recursive feasibility and closed-loop stability in the Lyapunov sense, that offers a more robust and effective approach to naval power and energy management. By comparing the proposed Lyapunov-based NMPC with conventional PI controller under fluctuating PPLs, the control robustness is validated.
Future Naval Microgrids (MGs) will include hybrid energy storage systems (ESS), including battery and supercapacitors to respond to emerging constant power loads (CPLs) and fluctuating pulsed power loads (PPLs). Voltage regulation of naval microgrids and power sharing among these resources become critical for success of a mission. This paper presents a novel control strategy using nonlinear model predictive controller embedded with a complex droop control architecture for voltage restoration and power sharing in medium voltage DC (MVDC) Naval MGs. The complex droop control ensures allocating supercapacitors (SCs) for high-frequency loads (i.e., PPLs), while battery energy storage system (BESS) and auxiliary generators share the steady-state load (i.e., CPL). Compared to state-of-the-art control of the naval ship MGs that relies on linear models, the proposed method incorporates the nonlinear behavior of the MGs in the closed-loop control framework via nonlinear model predictive control (NMPC). A reduced order representation of the MVDC dynamic is employed as the prediction model, augmented with a multi-objective, constraints-based, optimal control formulation. The results demonstrate the effectiveness of the proposed control framework for voltage restoration and power sharing of resources in naval MGs.
Cybersecurity in building energy management is crucial for protecting infrastructure, ensuring data integrity, and preventing unauthorized access or manipulation. This paper investigates the energy efficiency and cybersecurity of building energy management systems (BMS) against false data injection (FDI) attacks using proportional-integral (PI) and model predictive control (MPC) methods. Focusing on a commercial building model with five rooms, vulnerability of PI-based BMS and nonlinear MPC-based BMS against FDIs on sensors and actuators is studied. The study aims to assess the effectiveness of these control strategies in maintaining system performance and lifespan, highlighting the potential of MPC in enhancing system resilience against cyber threats. Our case studies demonstrate that even a short term FDIA can cause a 12% reduction in lifetime of a heat-pump under an MPC controller, and cause a near thirty-fold overuse of flow valves under a PI controller.
To reduce global carbon emissions and limit climate change, controlling energy consumption in buildings is an important piece of the puzzle. Here, we specifically focus on using a demand response (DR) algorithm to limit the energy consumption of a residential building's heating system while respecting user's thermal comfort. In that domain, Reinforcement learning (RL) methods have been shown to be quite effective. One such RL method is Monte Carlo Tree Search (MCTS), which has achieved impressive success in playing board games (go, chess). A particular advantage of MCTS is that its decision tree structure naturally allows to integrate exogenous constraints (e.g., by trimming branches that violate them), while conventional RL solutions need more elaborate techniques (e.g., indirectly by adding penalties in the cost/reward function, or through a backup controller that corrects constraint-violating actions). The main aim of this paper is to study the adoption of MCTS for building control, since this (to the best of our knowledge) has remained largely unexplored. A specific property of MCTS is that it needs a simulator component that can predict subsequent system states, based on actions taken. A straightforward data-driven solution is to use black-box neural networks (NNs). We will however extend a Physics-informed Neural Network (PiNN) model to deliver multi-timestep predictions, and show the benefit it offers in terms of lower prediction errors ($-$32\% MAE) as well as better MCTS performance ($-$4\% energy cost, $+$7\% thermal comfort) compared to a black-box NN. A second contribution will be to extend a vanilla MCTS version to adopt the ideas applied in AlphaZero (i.e., using learned prior and value functions and an action selection heuristic) to obtain lower computational costs while maintaining control performance.
Bioprocesses are often characterized by nonlinear and uncertain dynamics. This poses particular challenges in the context of model predictive control (MPC). Several approaches have been proposed to solve this problem, such as robust or stochastic MPC, but they can be computationally expensive when the system is nonlinear. Recent advances in optimal control theory have shown that concepts from convex optimization, tube-based MPC, and difference of convex functions (DC) enable stable and robust online process control. The approach is based on systematic DC decompositions of the dynamics and successive linearizations around feasible trajectories. By convexity, the linearization errors can be bounded tightly and treated as bounded disturbances in a robust tube-based MPC framework. However, finding the DC composition can be a difficult task. To overcome this problem, we used a neural network with special convex structure to learn the dynamics in DC form and express the uncertainty sets using simplices to maximize the product formation rate of a cultivation with uncertain substrate concentration in the feed. The results show that this is a promising approach for computationally tractable data-driven robust MPC of bioprocesses.
Autonomous Driving (AD) encounters significant safety hurdles in long-tail unforeseen driving scenarios, largely stemming from the non-interpretability and poor generalization of the deep neural networks within the AD system, particularly in out-of-distribution and uncertain data. To this end, this paper explores the integration of Large Language Models (LLMs) into AD systems, leveraging their robust common-sense knowledge and reasoning abilities. The proposed methodologies employ LLMs as intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning, for enhancing driving performance and safety. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine. Demonstrating superior performance and safety metrics compared to state-of-the-art approaches, our approach shows the promising potential for using LLMs for autonomous vehicles.
We address the challenge of dimension reduction in the discrete-time optimal control problem which is solved repeatedly online within the framework of model predictive control. Our study demonstrates that a reduced-order approach, aimed at identifying a suboptimal solution within a low-dimensional subspace, retains the stability and recursive feasibility characteristics of the original problem. We present a necessary and sufficient condition for ensuring initial feasibility, which is seamlessly integrated into the subspace design process. Additionally, we employ techniques from optimization on Riemannian manifolds to develop a subspace that efficiently represents a collection of pre-specified high-dimensional data points, all while adhering to the initial admissibility constraint.
For large uncertain systems, solving model predictive control problems online can be computationally taxing. Using a shorter prediction horizon can help, but may lead to poor performance and instability without appropriate modifications. This work focuses on learning convex objective terms to enable a single-step control horizon, reducing online computational costs. We consider two surrogates for approximating the cost-to-go: (1) a convex interpolating function and (2) an input-convex neural network. Regardless of the surrogate choice, its behavior near the origin and its ability to describe the feasible region are crucial for the closed-loop performance of the new MPC problem. We address this by tailoring the surrogate to ensure good performance in both aspects. We conclude with numerical examples, in which we compare the convex surrogates to using a standard neural network in the objective, solely using an LQR cost-to-go, and to using a neural network to learn a control policy. The proposed approaches are shown to achieve better performance with less data.
Humanoid control is an important research challenge offering avenues for integration into human-centric infrastructures and enabling physics-driven humanoid animations. The daunting challenges in this field stem from the difficulty of optimizing in high-dimensional action spaces and the instability introduced by the bipedal morphology of humanoids. However, the extensive collection of human motion-captured data and the derived datasets of humanoid trajectories, such as MoCapAct, paves the way to tackle these challenges. In this context, we present Humanoid Generalist Autoencoding Planner (H-GAP), a state-action trajectory generative model trained on humanoid trajectories derived from human motion-captured data, capable of adeptly handling downstream control tasks with Model Predictive Control (MPC). For 56 degrees of freedom humanoid, we empirically demonstrate that H-GAP learns to represent and generate a wide range of motor behaviours. Further, without any learning from online interactions, it can also flexibly transfer these behaviors to solve novel downstream control tasks via planning. Notably, H-GAP excels established MPC baselines that have access to the ground truth dynamics model, and is superior or comparable to offline RL methods trained for individual tasks. Finally, we do a series of empirical studies on the scaling properties of H-GAP, showing the potential for performance gains via additional data but not computing. Code and videos are available at https://ycxuyingchen.github.io/hgap/.
Effective control requires knowledge of the process dynamics to guide the system toward desired states. In many control applications this knowledge is expressed mathematically or through data-driven models, however, as complexity grows obtaining a satisfactory mathematical representation is increasingly difficult. Further, many data-driven approaches consist of abstract internal representations that may have no obvious connection to the underlying dynamics and control, or, require extensive model design and training. Here, we remove these constraints by demonstrating model predictive control from generalized state space embedding of the process dynamics providing a data-driven, explainable method for control of nonlinear, complex systems. Generalized embedding and model predictive control are demonstrated on nonlinear dynamics generated by an agent based model of 1200 interacting agents. The method is generally applicable to any type of controller and dynamic system representable in a state space.
The accurate prediction of smooth steering inputs is crucial for automotive applications because control actions with jitter might cause the vehicle system to become unstable. To address this problem in automobile lane-keeping control without the use of additional smoothing algorithms, we developed a novel soft-constrained iterative linear quadratic regulator (soft-CILQR) algorithm by integrating CILQR algorithm and a model predictive control (MPC) constraint relaxation method. We incorporated slack variables into the state and control barrier functions of the soft-CILQR solver to soften the constraints in the optimization process such that control input stabilization can be achieved in a computationally simple manner. Two types of automotive lane-keeping experiments (numerical simulations and experiments involving challenging vision-based maneuvers) were conducted with a linear system dynamics model to test the performance of the proposed soft-CILQR algorithm, and its performance was compared with that of the CILQR algorithm. In the numerical simulations, the soft-CILQR and CILQR solvers managed to drive the system toward the reference state asymptotically; however, the soft-CILQR solver obtained smooth steering input trajectories more easily than did the CILQR solver under conditions involving additive disturbances. The results of the vision-based experiments in which an ego vehicle drove in perturbed TORCS environments with various road friction settings were consistent with those of the numerical tests. The proposed soft-CILQR algorithm achieved an average runtime of 2.55 ms and is thus applicable for real-time autonomous driving scenarios.
This paper develops a sequencing-enabled hierarchical connected automated vehicle (CAV) cooperative on-ramp merging control framework. The proposed framework consists of a two-layer design: the upper level control sequences the vehicles to harmonize the traffic density across mainline and on-ramp segments while enhancing lower-level control efficiency through a mixed-integer linear programming formulation. Subsequently, the lower-level control employs a longitudinal distributed model predictive control (MPC) supplemented by a virtual car-following (CF) concept to ensure asymptotic local stability, l_2 norm string stability, and safety. Proofs of asymptotic local stability and l_2 norm string stability are mathematically derived. Compared to other prevalent asymptotic local-stable MPC controllers, the proposed distributed MPC controller greatly expands the initial feasible set. Additionally, an auxiliary lateral control is developed to maintain lane-keeping and merging smoothness while accommodating ramp geometric curvature. To validate the proposed framework, multiple numerical experiments are conducted. Results indicate a notable outperformance of our upper-level controller against a distance-based sequencing method. Furthermore, the lower-level control effectively ensures smooth acceleration, safe merging with adequate spacing, adherence to proven longitudinal local and string stability, and rapid regulation of lateral deviations.
We propose an integrated behavior and motion planning framework for the lane-merging problem. The behavior planner combines search-based planning with game theory to model vehicle interactions and plan multi-vehicle trajectories. Inspired by human drivers, we model the lane-merging problem as a gap selection process and determine the appropriate gap by solving a matrix game. Moreover, we introduce a branch model predictive control (BMPC) framework to account for the uncertain equilibrium strategies adopted by the surrounding vehicles, including Nash and Stackelberg strategies. A tailored numerical solver is developed to enhance computational efficiency by exploiting the tree structure inherent in BMPC. Finally, we validate our proposed integrated planner using real traffic data and demonstrate its effectiveness in handling interactions in dense traffic scenarios. The code is publicly available at: https://github.com/SailorBrandon/GT-BMPC.