Markov Decision Processes (MDPs) offer a fairly generic and powerful framework to discuss the notion of optimal policies for dynamic systems, in particular when the dynamics are stochastic. However, computing the optimal policy of an MDP can be very difficult due to the curse of dimensionality present in solving the underlying Bellman equations. Model Predictive Control (MPC) is a very popular technique for building control policies for complex dynamic systems. Historically, MPC has focused on constraint satisfaction and steering dynamic systems towards a user-defined reference. More recently, Economic MPC was proposed as a computationally tractable way of building optimal policies for dynamic systems. When stochsaticity is present, economic MPC is close to the MDP framework. In that context, Economic MPC can be construed as attractable heuristic to provide approximate solutions to MDPs. However, there is arguably a knowledge gap in the literature regarding these approximate solutions and the conditions for an MPC scheme to achieve closed-loop optimality. This chapter aims to clarify this approximation pedagogically, to provide the conditions for MPC to deliver optimal policies, and to explore some of their consequences.
Despite the availability of international prize-money competitions, scaled vehicles, and simulation environments, research on autonomous racing and the control of sports cars operating close to the limit of handling has been limited by the high costs of vehicle acquisition and management, as well as the limited physics accuracy of open-source simulators. In this paper, we propose a racing simulation platform based on the simulator Assetto Corsa to test, validate, and benchmark autonomous driving algorithms, including reinforcement learning (RL) and classical Model Predictive Control (MPC), in realistic and challenging scenarios. Our contributions include the development of this simulation platform, several state-of-the-art algorithms tailored to the racing environment, and a comprehensive dataset collected from human drivers. Additionally, we evaluate algorithms in the offline RL setting. All the necessary code (including environment and benchmarks), working examples, datasets, and videos are publicly released and can be found at: https://assetto-corsa-gym.github.io
Reinforcement learning (RL) and model predictive control (MPC) each offer distinct advantages and limitations when applied to control problems in power and energy systems. Despite various studies on these methods, benchmarks remain lacking and the preference for RL over traditional controls is not well understood. In this work, we put forth a comparative analysis using RL- and MPC-based controllers for optimizing a battery management system (BMS). The BMS problem aims to minimize costs while adhering to operational limits. by adjusting the battery (dis)charging in response to fluctuating electricity prices over a time horizon. The MPC controller uses a learningbased forecast of future demand and price changes to formulate a multi-period linear program, that can be solved using off-the-shelf solvers. Meanwhile, the RL controller requires no timeseries modeling but instead is trained from the sample trajectories using the proximal policy optimization (PPO) algorithm. Numerical tests compare these controllers across optimality, training time, testing time, and robustness, providing a comprehensive evaluation of their efficacy. RL not only yields optimal solutions quickly but also ensures robustness to shifts in customer behavior, such as changes in demand distribution. However, as expected, training the RL agent is more time-consuming than MPC.
Scenario-based optimization and control has proven to be an efficient approach to account for system uncertainty. In particular, the performance of scenario-based model predictive control (MPC) schemes depends on the accuracy of uncertainty quantification. However, current learning- and scenario-based MPC (sMPC) approaches employ a single timeinvariant probabilistic model (learned offline), which may not accurately describe time-varying uncertainties. Instead, this paper presents a model-agnostic meta-learning (MAML) of Bayesian neural networks (BNN) for adaptive uncertainty quantification that would be subsequently used for adaptive-scenario-tree model predictive control design of nonlinear systems with unknown dynamics to enhance control performance. In particular, the proposed approach learns both a global BNN model and an updating law to refine the BNN model. At each time step, the updating law transforms the global BNN model into more precise local BNN models in real time. The adapted local model is then used to generate scenarios for sMPC design at each time step. A probabilistic safety certificate is incorporated in the scenario generation to ensure that the trajectories of the generated scenarios contain the real trajectory of the system and that all the scenarios adhere to the constraints with a high probability. Experiments using closed-loop simulations of a numerical example demonstrate that the proposed approach can improve the performance of scenario-based MPC compared to using only one BNN model learned offline for all time steps.
A major challenge in the development of Model Predictive Control (MPC)-based energy management systems (EMSs) for buildings is the availability of an accurate model. One approach to address this is to augment an existing gray-box model with data-driven residual estimators. The efficacy of such estimators, and hence the performance of the EMS, relies on the availability of sufficient and suitable training data. In this work, we evaluate how different data availability scenarios affect estimator and controller performance. To do this, we perform software-in-the-loop (SiL) simulation with a physics-based digital twin using real measurement data. Simulation results show that acceptable estimation and control performance can already be achieved with limited available data, and we confirm that leveraging historical data for pretraining boosts efficacy.
We present a stochastic model predictive control framework for nonlinear systems subject to unbounded process noise with closed-loop guarantees. First, we provide a conceptual shrinking-horizon framework that utilizes general probabilistic reachable sets and minimizes the expected cost. Then, we provide a tractable receding-horizon formulation that uses a nominal state to minimize a deterministic quadratic cost and satisfy tightened constraints. Our theoretical analysis demonstrates recursive feasibility, satisfaction of chance constraints, and bounds on the expected cost for the resulting closed-loop system. We provide a constructive design for probabilistic reachable sets of nonlinear continuously differentiable systems using stochastic contraction metrics and an assumed bound on the covariance matrices. Numerical simulations highlight the computational efficiency and theoretical guarantees of the proposed method. Overall, this paper provides a framework for computationally tractable stochastic predictive control with closed-loop guarantees for nonlinear systems with unbounded noise.
Model predictive control (MPC) has played a more crucial role in various robotic control tasks, but its high computational requirements are concerning, especially for nonlinear dynamical models. This paper presents a $\textbf{la}$tent $\textbf{l}$inear $\textbf{q}$uadratic $\textbf{r}$egulator (LaLQR) that maps the state space into a latent space, on which the dynamical model is linear and the cost function is quadratic, allowing the efficient application of LQR. We jointly learn this alternative system by imitating the original MPC. Experiments show LaLQR's superior efficiency and generalization compared to other baselines.
This study addresses the challenges of energy deficiencies and high impact low probability (HILP) events in modern electrical grids by developing resilient microgrid energy management strategies. It introduces a sliding Model Predictive Control (MPC) methodology integrated with Battery Energy Storage Systems (BESS), emphasizing extending battery life and prioritizing critical loads during HILP events. This approach focuses on extending the sustainability of battery operation by linearizing the battery lifecycle within the optimization framework. Furthermore, this research proposed a straightforward method to mitigate communication disruptions during HILP events, thereby ensuring operational integrity. This focused approach enhances isolated microgrid resilience and sustainability, offering a strategic response to contemporary environmental challenges.
This paper presents the open-source stochastic model predictive control framework GRAMPC-S for nonlinear uncertain systems with chance constraints. It provides several uncertainty propagation methods to predict stochastic moments of the system state and can consider unknown parts of the system dynamics using Gaussian process regression. These methods are used to reformulate a stochastic MPC formulation as a deterministic one that is solved with GRAMPC. The performance of the presented framework is evaluated using examples from a wide range of technical areas. The experimental evaluation shows that GRAMPC-S can be used in practice for the control of nonlinear uncertain systems with sampling times in the millisecond range.
With the development of autonomous driving technology, there are increasing demands for vehicle control, and MPC has become a widely researched topic in both industry and academia. Existing MPC control methods based on vehicle kinematics or dynamics have challenges such as difficult modeling, numerous parameters, strong nonlinearity, and high computational cost. To address these issues, this paper adapts an existing Data-driven MPC control method and applies it to autonomous vehicle steering control. This method avoids the need for complex vehicle system modeling and achieves trajectory tracking with relatively low computational time and small errors. We validate the control effectiveness of the algorithm in specific scenario through CarSim-Simulink simulation and perform comparative analysis with PID and vehicle kinematics MPC, confirming the feasibility and superiority of it for vehicle steering control.
Multi-agent systems (MAS) constitute a significant role in exploring machine intelligence and advanced applications. In order to deeply investigate complicated interactions within MAS scenarios, we originally propose "GNN for MBRL" model, which utilizes a state-spaced Graph Neural Networks with Model-based Reinforcement Learning to address specific MAS missions (e.g., Billiard-Avoidance, Autonomous Driving Cars). In detail, we firstly used GNN model to predict future states and trajectories of multiple agents, then applied the Cross-Entropy Method (CEM) optimized Model Predictive Control to assist the ego-agent planning actions and successfully accomplish certain MAS tasks.
Nonlinear model predictive control (NMPC) has proven to be an effective control method, but it is expensive to compute. This work demonstrates the use of hardware FPGA neural network controllers trained to imitate NMPC with supervised learning. We use these Neural Controllers (NCs) implemented on inexpensive embedded FPGA hardware for high frequency control on physical cartpole and F1TENTH race car. Our results show that the NCs match the control performance of the NMPCs in simulation and outperform it in reality, due to the faster control rate that is afforded by the quick FPGA NC inference. We demonstrate kHz control rates for a physical cartpole and offloading control to the FPGA hardware on the F1TENTH car. Code and hardware implementation for this paper are available at https:// github.com/SensorsINI/Neural-Control-Tools.
According to a conservative estimate, a 1% reduction in forecast error for a 10 GW energy utility can save up to $ 1.6 million annually. In our context, achieving precise forecasts of future power consumption is crucial for operating flexible energy assets using model predictive control approaches. Specifically, this work focuses on the load profile forecast of a first-year energy community with the common practical challenge of limited historical data availability. We propose to pre-train the load prediction models with open-access synthetic load profiles using transfer learning techniques to tackle this challenge. Results show that this approach improves both, the training stability and prediction error. In a test case with 74 households, the prediction mean squared error (MSE) decreased from 0.34 to 0.13, showing transfer learning based on synthetic load profiles to be a viable approach to compensate for a lack of historic data.
Foundation models, e.g., large language models (LLMs), trained on internet-scale data possess zero-shot generalization capabilities that make them a promising technology towards detecting and mitigating out-of-distribution failure modes of robotic systems. Fully realizing this promise, however, poses two challenges: (i) mitigating the considerable computational expense of these models such that they may be applied online, and (ii) incorporating their judgement regarding potential anomalies into a safe control framework. In this work, we present a two-stage reasoning framework: First is a fast binary anomaly classifier that analyzes observations in an LLM embedding space, which may then trigger a slower fallback selection stage that utilizes the reasoning capabilities of generative LLMs. These stages correspond to branch points in a model predictive control strategy that maintains the joint feasibility of continuing along various fallback plans to account for the slow reasoner's latency as soon as an anomaly is detected, thus ensuring safety. We show that our fast anomaly classifier outperforms autoregressive reasoning with state-of-the-art GPT models, even when instantiated with relatively small language models. This enables our runtime monitor to improve the trustworthiness of dynamic robotic systems, such as quadrotors or autonomous vehicles, under resource and time constraints. Videos illustrating our approach in both simulation and real-world experiments are available on this project page: https://sites.google.com/view/aesop-llm.
In many state-of-the-art control approaches for power systems with storage units, an explicit model of the storage dynamics is required. With growing numbers of storage units, identifying these dynamics can be cumbersome. This paper employs recent data-driven control approaches that do not require an explicit identification step. Instead, they use measured input/output data in control formulations. In detail, we propose an economic data-driven model predictive control (MPC) scheme to operate a small power system with input-nonlinear battery dynamics. First, a linear data-driven MPC approach that uses a slack variable to account for plant-model-mismatch is proposed. In a second step, an input-nonlinear data-driven MPC scheme is deduced. Comparisons with a reference indicate that the linear data-driven MPC approximates the nonlinear plant in an acceptable manner. Even better results, however, can be obtained with the input-nonlinear data-driven MPC scheme which provides increased prediction accuracy.
Model Predictive Control (MPC) is an optimal control strategy suited for flood control of water resources infrastructure. Despite many studies on reservoir flood control and their theoretical contribution, optimisation methodologies have not been widely applied in real-time operation due to disparities between research assumptions and practical requirements. To address this gap, we include practical objectives, such as minimising the magnitude and frequency of changes in the existing outflow schedule. Incorporating these objectives transforms the problem into a multi-objective nonlinear optimisation problem that is difficult to solve in real-time. Additionally, it is reasonable to assume that the weights and some parameters, considered the operators' preferences, vary depending on the system state. To overcome these limitations, we propose a framework that converts the original intractable problem into parameterized linear MPC problems with dynamic optimisation of weights and parameters. This is done by introducing a model-based learning concept. We refer to this framework as Parameterised Dynamic MPC (PD-MPC). The effectiveness of this framework is demonstrated through a numerical experiment for the Daecheong multipurpose reservoir in South Korea. We find that PD-MPC outperforms standard MPC-based designs without a dynamic optimisation process for the objective weights and model parameters. Moreover, we demonstrate that the weights and parameters vary with changing hydrological conditions.
We present a Learning Model Predictive Controller (LMPC) for multi-modal systems performing iterative control tasks. Assuming availability of historical data, our goal is to design a data-driven control policy for the multi-modal system where the current mode is unknown. First, we propose a novel method to select local data for constructing affine time-varying (ATV) models of a multi-modal system in the context of LMPC. Then we present how to build a sampled safe set from multi-modal historical data. We demonstrate the effectiveness of our method through simulation results of automated driving on a friction-varying track.
Traditional prediction-dependent dispatch methods can face challenges when renewables and prices predictions are unreliable in microgrid. Instead, this paper proposes a novel prediction-free two-stage coordinated dispatch approach in microgrid. Empirical learning is conducted during the offline stage, where we calculate the offline optimal state of charge (SOC) sequences for generic energy storage under different historical scenarios. During the online stage, we synthesize a dynamically updated reference for SOC and a dynamic opportunity price (DOP) based on empirical learning and real-time observations. They provide a global vision for online operation and effectively address the myopic tendencies inherent to online decision-making. The real-time control action, generated from online optimization algorithm, aims to minimize the operational costs while tracking the reference and considering DOP. Additionally, we develop an adaptive virtual-queue-based online optimization algorithm based on online convex optimization (OCO) framework. We provide theoretical proof that the proposed algorithm outperforms the existing OCO algorithms and achieves sublinear dynamic regret bound and sublinear strict constraint violation bound. Simulation-based studies demonstrate that, compared with model predictive control-based methods, it reduces operational costs and voltage violation rate by 5% and 9%, respectively.
Model predictive control of residential air conditioning could reduce energy costs and greenhouse gas emissions while maintaining or improving occupants' thermal comfort. However, most approaches to predictive air conditioning control either do not model indoor humidity or treat it as constant. This simplification stems from challenges with modeling indoor humidity dynamics, particularly the high-order, nonlinear equations that govern heat and mass transfer between the air conditioner's evaporator coil and the indoor air. This paper develops a machine-learning approach to modeling indoor humidity dynamics that is suitable for real-world deployment at scale. This study then investigates the value of humidity modeling in four field tests of predictive control in an occupied house. The four field tests evaluate two different building models: One with constant humidity and one with time-varying humidity. Each modeling approach is tested in two different predictive controllers: One that focuses on reducing energy costs and one that focuses on constraining electric power below a utility-specified threshold. The two models lead to similar performance for reducing energy costs. Combining the results of this study and a prior heating study of the same house, the estimated year-round energy cost savings were $340-497 or 22-31% (95% confidence intervals); these savings were consistent across both humidity models. However, in the demand response tests, the simplifying assumption of constant humidity led to far more frequent and severe violations of the power constraint. These results suggest that accurate building models are important for nonlinear objectives, such as reducing or constraining peak demand, while for linear objectives such as reducing energy costs or emissions, model accuracy is less important.
Building on our previous work, this paper investigates the effectiveness of interpolating control (IC) for real-time trajectory tracking. Unlike prior studies that focused on trajectory tracking itself or UAV stabilization control in simulation, we evaluate the performance of a modified extended IC (eIC) controller compared to Model Predictive Control (MPC) through both simulated and laboratory experiments with a remotely controlled UAV. The evaluation focuses on the computational efficiency and control quality of real-time UAV trajectory tracking compared to previous IC applications. The results demonstrate that the eIC controller achieves competitive performance compared to MPC while significantly reducing computational complexity, making it a promising alternative for resource-constrained platforms.