Mobile robots should be capable of planning cost-efficient paths for autonomous navigation. Typically, the terrain and robot properties are subject to variations. For instance, properties of the terrain such as friction may vary across different locations. Also, properties of the robot may change such as payloads or wear and tear, e.g., causing changing actuator gains or joint friction. Autonomous navigation approaches should thus be able to adapt to such variations. In this article, we propose a novel approach for learning a probabilistic, terrain- and robot-aware forward dynamics model (TRADYN) which can adapt to such variations and demonstrate its use for navigation. Our learning approach extends recent advances in meta-learning forward dynamics models based on Neural Processes for mobile robot navigation. We evaluate our method in simulation for 2D navigation of a robot with uni-cycle dynamics with varying properties on terrain with spatially varying friction coefficients. In our experiments, we demonstrate that TRADYN has lower prediction error over long time horizons than model ablations which do not adapt to robot or terrain variations. We also evaluate our model for navigation planning in a model-predictive control framework and under various sources of noise. We demonstrate that our approach yields improved performance in planning control-efficient paths by taking robot and terrain properties into account.
This work proposes an approach that integrates reinforcement learning and model predictive control (MPC) to solve finite-horizon optimal control problems in mixed-logical dynamical systems efficiently. Optimization-based control of such systems with discrete and continuous decision variables entails the online solution of mixed-integer linear programs, which suffer from the curse of dimensionality. Our approach aims to mitigate this issue by decoupling the decision on the discrete variables from the decision on the continuous variables. In the proposed approach, reinforcement learning determines the discrete decision variables and simplifies the online optimization problem of the MPC controller from a mixed-integer linear program to a linear program, significantly reducing the computational time. A fundamental contribution of this work is the definition of the decoupled Q-function, which plays a crucial role in making the learning problem tractable in a combinatorial action space. We motivate the use of recurrent neural networks to approximate the decoupled Q-function and show how they can be employed in a reinforcement learning setting. Simulation experiments on a microgrid system using real-world data demonstrate that the proposed method substantially reduces the online computation time of MPC while maintaining high feasibility and low suboptimality.
Safety remains a central challenge in control of dynamical systems, particularly when the boundaries of unsafe sets are complex or unknown. This paper proposes a learning-enabled framework for safety-critical Model Predictive Control (MPC) that integrates Discrete-Time High-Order Control Barrier Functions (DHOCBFs) with iterative convex optimization. Unlike existing methods that primarily address CBFs of relative degree one with fully known unsafe set boundaries, our approach generalizes to arbitrary relative degrees and addresses scenarios where the unsafe set boundaries must be inferred. We extract pixel-based data specifically from unsafe set boundaries and train a neural network to approximate local linearizations of these boundaries. The learned models are incorporated into the linearized DHOCBF constraints at each time step, enabling real-time constraint satisfaction within the MPC framework. An iterative convex optimization procedure is developed to accelerate computation while maintaining formal safety guarantees. The benefits of computational performance and safe avoidance of obstacles with diverse shapes are examined and confirmed through numerical results. By bridging model-based control with learning-based environment modeling, this framework advances safe autonomy for discrete-time systems operating in complex and partially known settings.
We propose a novel data-driven stochastic model predictive control framework for uncertain linear systems with noisy output measurements. Our approach leverages multi-step predictors to efficiently propagate uncertainty, ensuring chance constraint satisfaction. In particular, we present a strategy to identify multi-step predictors and quantify the associated uncertainty using a surrogate (data-driven) state space model. Then, we utilize the derived distribution to formulate a constraint tightening that ensures chance constraint satisfaction despite the parametric uncertainty. A numerical example highlights the reduced conservatism of handling parametric uncertainty in the proposed method compared to state-of-the-art solutions.
In the rapidly evolving field of vision-language navigation (VLN), ensuring safety for physical agents remains an open challenge. For a human-in-the-loop language-operated drone to navigate safely, it must understand natural language commands, perceive the environment, and simultaneously avoid hazards in real time. Control Barrier Functions (CBFs) are formal methods that enforce safe operating conditions. Model Predictive Control (MPC) is an optimization framework that plans a sequence of future actions over a prediction horizon, ensuring smooth trajectory tracking while obeying constraints. In this work, we consider a VLN-operated drone platform and enhance its safety by formulating a novel scene-aware CBF that leverages ego-centric observations from a camera which has both Red-Green-Blue as well as Depth (RGB-D) channels. A CBF-less baseline system uses a Vision-Language Encoder with cross-modal attention to convert commands into an ordered sequence of landmarks. An object detection model identifies and verifies these landmarks in the captured images to generate a planned path. To further enhance safety, an Adaptive Safety Margin Algorithm (ASMA) is proposed. ASMA tracks moving objects and performs scene-aware CBF evaluation on-the-fly, which serves as an additional constraint within the MPC framework. By continuously identifying potentially risky observations, the system performs prediction in real time about unsafe conditions and proactively adjusts its control actions to maintain safe navigation throughout the trajectory. Deployed on a Parrot Bebop2 quadrotor in the Gazebo environment using the Robot Operating System (ROS), ASMA achieves 64%-67% increase in success rates with only a slight increase (1.4%-5.8%) in trajectory lengths compared to the baseline CBF-less VLN.
Distributed control algorithms are known to reduce overall computation time compared to centralized control algorithms. However, they can result in inconsistent solutions leading to the violation of safety-critical constraints. Inconsistent solutions can arise when two or more agents compute concurrently while making predictions on each others control actions. To address this issue, we propose an iterative algorithm called Synchronization-Based Cooperative Distributed Model Predictive Control, which we presented in [1]. The algorithm consists of two steps: 1. computing the optimal control inputs for each agent and 2. synchronizing the predicted states across all agents. We demonstrate the efficacy of our algorithm in the control of multiple small-scale vehicles in our Cyber-Physical Mobility Lab.
As seasonal thermal energy storage emerges as an efficient solution to reduce CO2 emissions of buildings, challenges appear related to its optimal operation. In a system including short-term electricity storage, long-term heat storage, and where electricity and heat networks are connected through a heat pump, it becomes crucial to operate the system on two time scales. Based on real data from a university building, we simulate the operation of such a system over a year, comparing different strategies based on model predictive control (MPC). The first objective of this paper is to determine the minimum prediction horizon to retrieve the results of the full-horizon operation problem with cost minimization. The second objective is to evaluate a method that combines MPC with setting targets on the heat storage level at the end of the prediction horizon, based on historical data. For a prediction horizon of 6 days, the suboptimality gap with the full-horizon results is 4.31%, compared to 11.42% when using a prediction horizon of 42 days and fixing the final level to be equal to the initial level, which is a common approach.
A core strength of Model Predictive Control (MPC) for quadrupedal locomotion has been its ability to enforce constraints and provide interpretability of the sequence of commands over the horizon. However, despite being able to plan, MPC struggles to scale with task complexity, often failing to achieve robust behavior on rapidly changing surfaces. On the other hand, model-free Reinforcement Learning (RL) methods have outperformed MPC on multiple terrains, showing emergent motions but inherently lack any ability to handle constraints or perform planning. To address these limitations, we propose a framework that integrates proprioceptive planning with RL, allowing for agile and safe locomotion behaviors through the horizon. Inspired by MPC, we incorporate an internal model that includes a velocity estimator and a Dreamer module. During training, the framework learns an expert policy and an internal model that are co-dependent, facilitating exploration for improved locomotion behaviors. During deployment, the Dreamer module solves an infinite-horizon MPC problem, adapting actions and velocity commands to respect the constraints. We validate the robustness of our training framework through ablation studies on internal model components and demonstrate improved robustness to training noise. Finally, we evaluate our approach across multi-terrain scenarios in both simulation and hardware.
Learning uncertain dynamics models using Gaussian process~(GP) regression has been demonstrated to enable high-performance and safety-aware control strategies for challenging real-world applications. Yet, for computational tractability, most approaches for Gaussian process-based model predictive control (GP-MPC) are based on approximations of the reachable set that are either overly conservative or impede the controller's safety guarantees. To address these challenges, we propose a robust GP-MPC formulation that guarantees constraint satisfaction with high probability. For its tractable implementation, we propose a sampling-based GP-MPC approach that iteratively generates consistent dynamics samples from the GP within a sequential quadratic programming framework. We highlight the improved reachable set approximation compared to existing methods, as well as real-time feasible computation times, using two numerical examples.
Safe learning of control policies remains challenging, both in optimal control and reinforcement learning. In this article, we consider safe learning of parametrized predictive controllers that operate with incomplete information about the underlying process. To this end, we employ Bayesian optimization for learning the best parameters from closed-loop data. Our method focuses on the system's overall long-term performance in closed-loop while keeping it safe and stable. Specifically, we parametrize the stage cost function of an MPC using a feedforward neural network. This allows for a high degree of flexibility, enabling the system to achieve a better closed-loop performance with respect to a superordinate measure. However, this flexibility also necessitates safety measures, especially with respect to closed-loop stability. To this end, we explicitly incorporated stability information in the Bayesian-optimization-based learning procedure, thereby achieving rigorous probabilistic safety guarantees. The proposed approach is illustrated using a numeric example.
This work presents and showcases a novel reinforcement learning agent called Critic As Lyapunov Function (CALF) which is model-free and ensures online environment, in other words, dynamical system stabilization. Online means that in each learning episode, the said environment is stabilized. This, as demonstrated in a case study with a mobile robot simulator, greatly improves the overall learning performance. The base actor-critic scheme of CALF is analogous to SARSA. The latter did not show any success in reaching the target in our studies. However, a modified version thereof, called SARSA-m here, did succeed in some learning scenarios. Still, CALF greatly outperformed the said approach. CALF was also demonstrated to improve a nominal stabilizer provided to it. In summary, the presented agent may be considered a viable approach to fusing classical control with reinforcement learning. Its concurrent approaches are mostly either offline or model-based, like, for instance, those that fuse model-predictive control into the agent.
Control barrier functions (CBFs) offer an efficient framework for designing real-time safe controllers. However, CBF-based controllers can be short-sighted, resulting in poor performance, a behaviour which is aggravated in uncertain conditions. This motivated research on safety filters based on model predictive control (MPC) and its stochastic variant. MPC deals with safety constraints in a direct manner, however, its computational demands grow with the prediction horizon length. We propose a safety formulation that solves a finite horizon optimization problem at each time instance like MPC, but rather than explicitly imposing constraints along the prediction horizon, we enforce probabilistic safety constraints by means of CBFs only at the first step of the horizon. The probabilistic CBF constraints are transformed in a finite number of deterministic CBF constraints via the scenario based methodology. Capitalizing on results on scenario based MPC, we provide distribution-free, \emph{a priori} guarantees on the system's closed loop expected safety violation frequency. We demonstrate our results through a case study on unmanned aerial vehicle collision-free position swapping, and provide a numerical comparison with recent stochastic CBF formulations.
This paper explores the use of Autoencoder (AE) models to identify Koopman-based linear representations for designing model predictive control (MPC) for wind farms. Wake interactions in wind farms are challenging to model, previously addressed with Koopman lifted states. In this study we investigate the performance of two AE models: The first AE model estimates the wind speeds acting on the turbines these are affected by changes in turbine control inputs. The wind speeds estimated by this AE model are then used in a second step to calculate the power output via a simple turbine model based on physical equations. The second AE model directly estimates the wind farm output, i.e., both turbine and wake dynamics are modeled. The primary inquiry of this study addresses whether any of these two AE-based models can surpass previously identified Koopman models based on physically motivated lifted states. We find that the first AE model, which estimates the wind speed and hence includes the wake dynamics, but excludes the turbine dynamics outperforms the existing physically motivated Koopman model. However, the second AE model, which estimates the farm power directly, underperforms when the turbines' underlying physical assumptions are correct. We additionally investigate specific conditions under which the second, purely data-driven AE model can excel: Notably, when modeling assumptions, such as the wind turbine power coefficient, are erroneous and remain unchecked within the MPC controller. In such cases, the data-driven AE models, when updated with recent data reflecting changed system dynamics, can outperform physics-based models operating under outdated assumptions.
Residential electrification - replacing fossil-fueled appliances and vehicles with electric machines - can significantly reduce greenhouse gas emissions and air pollution. However, installing electric appliances or vehicle charging in a residential building can sharply increase its current draws. In older housing, high current draws can jeopardize electrical infrastructure, such as circuit breaker panels or electrical service (the wires that connect a building to the distribution grid). Upgrading electrical infrastructure can entail long delays and high costs, so poses a significant barrier to electrification. This paper develops and field-tests a control system that avoids the need for electrical upgrades by keeping an electrified home's total current draw within the safe limits of its panel and service. In the proposed control architecture, a high-level controller plans device set-points over a rolling prediction horizon. A low-level controller monitors real-time conditions and ramps down devices if necessary. The control system was tested in an occupied, electrified single-family house with code-minimum insulation, an air-to-air heat pump and backup resistance heat, a resistance water heater, and a plug-in hybrid electric vehicle with Level I charging. The field tests spanned 31 winter days with outdoor temperatures as low as -20 C. The control system maintained the whole-home current within the safe limits of electrical panels and service rated at 100 A, a common rating for older houses in North America, by adjusting only the temperature set-points of the heat pump and water heater. Simulations suggest that the same 100 A limit could accommodate a second electric vehicle with Level II charging. The proposed control system could allow older homes to safely electrify without upgrading electrical panels or service, saving a typical household on the order of $2,000 to $10,000.
In this paper we apply model predictive control (MPC), rollout, and reinforcement learning (RL) methodologies to computer chess. We introduce a new architecture for move selection, within which available chess engines are used as components. One engine is used to provide position evaluations in an approximation in value space MPC/RL scheme, while a second engine is used as nominal opponent, to emulate or approximate the moves of the true opponent player. We show that our architecture improves substantially the performance of the position evaluation engine. In other words our architecture provides an additional layer of intelligence, on top of the intelligence of the engines on which it is based. This is true for any engine, regardless of its strength: top engines such as Stockfish and Komodo Dragon (of varying strengths), as well as weaker engines. Structurally, our basic architecture selects moves by a one-move lookahead search, with an intermediate move generated by a nominal opponent engine, and followed by a position evaluation by another chess engine. Simpler schemes that forego the use of the nominal opponent, also perform better than the position evaluator, but not quite by as much. More complex schemes, involving multistep lookahead, may also be used and generally tend to perform better as the length of the lookahead increases. Theoretically, our methodology relies on generic cost improvement properties and the superlinear convergence framework of Newton's method, which fundamentally underlies approximation in value space, and related MPC/RL and rollout/policy iteration schemes. A critical requirement of this framework is that the first lookahead step should be executed exactly. This fact has guided our architectural choices, and is apparently an important factor in improving the performance of even the best available chess engines.
Robust model predictive control algorithms are essential for addressing unavoidable errors due to the uncertainty in predicting real-world systems. However, the formulation of such algorithms typically results in a trade-off between conservatism and computational complexity. Monotone systems facilitate the efficient computation of reachable sets and thus the straightforward formulation of a robust model predictive control approach optimizing over open-loop predictions. We present an approach based on the division of reachable sets to incorporate feedback in the predictions, resulting in less conservative strategies. The concept of mixed-monotonicity enables an extension of our methodology to non-monotone systems. The potential of the proposed approaches is demonstrated through a nonlinear high-dimensional chemical tank reactor cascade case study.
It is challenging to coordinate multiple distributed energy resources in a single or multiple buildings to ensure efficient and flexible operation. Advanced control algorithms such as model predictive control and reinforcement learning control provide solutions to this problem by effectively managing a distribution of distributed energy resource control tasks while adapting to unique building characteristics, and cooperating towards improving multi-objective key performance indicator. Yet, a research gap for advanced control adoption is the ability to benchmark algorithm performance. CityLearn addresses this gap an open-source Gym environment for the easy implementation and benchmarking of simple rule-based control and advanced algorithms that has an advantage of modeling simplicity, multi-agent control, district-level objectives, and control resiliency assessment. Here we demonstrate the functionalities of CityLearn using 17 different building control problems that have varying complexity with respect to the number of controllable distributed energy resources in buildings, the simplicity of the control algorithm, the control objective, and district size.
In this paper, a fixed-time disturbance observerbased model predictive control algorithm is proposed for trajectory tracking of quadrotor in the presence of disturbances. First, a novel multivariable fixed-time disturbance observer is proposed to estimate the lumped disturbances. The bi-limit homogeneity and Lyapunov techniques are employed to ensure the convergence of estimation error within a fixed convergence time, independent of the initial estimation error. Then, an observerbased model predictive control strategy is formulated to achieve robust trajectory tracking of quadrotor, attenuating the lumped disturbances and model uncertainties. Finally, simulations and real-world experiments are provided to illustrate the effectiveness of the proposed method.
In this paper, the model predictive control (MPC) problem is investigated for the constrained discrete-time Takagi-Sugeno fuzzy Markovian jump systems (FMJSs) under imperfect premise matching rules. To strike a balance between initial feasible region, control performance, and online computation burden, a set of mode-dependent state feedback fuzzy controllers within the frame of dynamic prediction optimizing (DPO)-MPC is delicately designed with the perturbation variables produced by the predictive dynamics. The DPO-MPC controllers are implemented via two stages: at the first stage, terminal constraints sets companied with feedback gain are obtained by solving a ``min-max'' problem; at the second stage, and a set of perturbations is designed felicitously to enlarge the feasible region. Here, dynamic feedback gains are designed for off-line using matrix factorization technique, while the dynamic controller state is determined for online over a moving horizon to gradually guide the system state from the initial feasible region to the terminal constraint set. Sufficient conditions are provided to rigorously ensure the recursive feasibility of the proposed DPO-MPC scheme and the mean-square stability of the underlying FMJS. Finally, the efficacy of the proposed methods is demonstrated through a robot arm system example.
High-precision displacement control for water-hydraulic artificial muscles is a challenging issue due to its strong hysteresis characteristics that is hard to be modelled precisely, and many control methods have been proposed. Recently, data-driven control methods have attracted much attention because they do not explicitly use mathematical models, making design much easier. In our previous work, we proposed fictitious reference iterative tuning (FRIT)-based model predictive control (FMPC), which combines data-driven and model-based methods for the muscle and showed its effectiveness because it can consider input constraints as well. However, the problem in which control performance strongly depends on prior input-output data remains still unsolved. Adaptive FRIT based on directional forgetting has also been proposed; however, it is difficult to achieve the desired transient performance because it cannot consider input constraints and there are no design parameters that directly determine the control performance, such as MPC. In this study, we propose a novel data-driven adaptive model matching-based controller that combines these methods. Experimental results show that the proposed method could significantly improve the control performance and achieve high robustness against inappropriate initial experimental data , while considering the input constraints in the design phase.