Distributed model predictive control (DMPC) is often used to tackle path planning for unmanned aerial vehicle (UAV) swarms. However, it requires considerable computations on-board the UAV, leading to increased weight and power consumption. In this work, we propose to offload path planning computations to multiple ground-based computation units. As simultaneously communicating and recomputing all trajectories is not feasible for a large swarm with tight timing requirements, we develop a novel event-triggered DMPC that selects a subset of most relevant UAV trajectories to be replanned. The resulting architecture reduces UAV weight and power consumption, while the active redundancy provides robustness against computation unit failures. Moreover, the DMPC guarantees feasible and collision-free trajectories for UAVs with linear dynamics. In simulations, we demonstrate that our method can reliably plan trajectories, while saving 60% of network traffic and required computational power. Hardware-in-the-loop experiments show that it is suitable to control real quadcopter swarms.
Traditional biological and pharmaceutical manufacturing plants are controlled by human workers or pre-defined thresholds. Modernized factories have advanced process control algorithms such as model predictive control (MPC). However, there is little exploration of applying deep reinforcement learning to control manufacturing plants. One of the reasons is the lack of high fidelity simulations and standard APIs for benchmarking. To bridge this gap, we develop an easy-to-use library that includes five high-fidelity simulation environments: BeerFMTEnv, ReactorEnv, AtropineEnv, PenSimEnv and mAbEnv, which cover a wide range of manufacturing processes. We build these environments on published dynamics models. Furthermore, we benchmark online and offline, model-based and model-free reinforcement learning algorithms for comparisons of follow-up research.
Accurately modeling quadrotor's system dynamics is critical for guaranteeing agile, safe, and stable navigation. The model needs to capture the system behavior in multiple flight regimes and operating conditions, including those producing highly nonlinear effects such as aerodynamic forces and torques, rotor interactions, or possible system configuration modifications. Classical approaches rely on handcrafted models and struggle to generalize and scale to capture these effects. In this paper, we present a novel Physics-Inspired Temporal Convolutional Network (PI-TCN) approach to learning quadrotor's system dynamics purely from robot experience. Our approach combines the expressive power of sparse temporal convolutions and dense feed-forward connections to make accurate system predictions. In addition, physics constraints are embedded in the training process to facilitate the network's generalization capabilities to data outside the training distribution. Finally, we design a model predictive control approach that incorporates the learned dynamics for accurate closed-loop trajectory tracking fully exploiting the learned model predictions in a receding horizon fashion. Experimental results demonstrate that our approach accurately extracts the structure of the quadrotor's dynamics from data, capturing effects that would remain hidden to classical approaches. To the best of our knowledge, this is the first time physics-inspired deep learning is successfully applied to temporal convolutional networks and to the system identification task, while concurrently enabling predictive control.
Model predictive controllers use dynamics models to solve constrained optimal control problems. However, computational requirements for real-time control have limited their use to systems with low-dimensional models. Nevertheless, high-dimensional models arise in many settings, for example discretization methods for generating finite-dimensional approximations to partial differential equations can result in models with thousands to millions of dimensions. In such cases, reduced order models (ROMs) can significantly reduce computational requirements, but model approximation error must be considered to guarantee controller performance. In this work, a reduced order model predictive control (ROMPC) scheme is proposed to solve robust, output feedback, constrained optimal control problems for high-dimensional linear systems. Computational efficiency is obtained by using projection-based ROMs, and guarantees on robust constraint satisfaction and stability are provided. Performance of the approach is demonstrated in simulation for several examples, including an aircraft control problem leveraging an inviscid computational fluid dynamics model with dimension 998,930.
Nonlinear model predictive control (NMPC) has gained widespread use in many applications. Its formulation traditionally involves repetitively solving a nonlinear constrained optimization problem online. In this paper, we investigate NMPC through the lens of Bayesian estimation and highlight that the Monte Carlo sampling method can offer a favorable way to implement NMPC. We develop a constraint-aware particle filtering/smoothing method and exploit it to implement NMPC. The new sampling-based NMPC algorithm can be executed easily and efficiently even for complex nonlinear systems, while potentially mitigating the issues of computational complexity and local minima faced by numerical optimization in conventional studies. The effectiveness of the proposed algorithm is evaluated through a simulation study.
The increasing access to non-expensive sensors, computing power, and more accurate forecasting of storm events provides unique opportunities to shift flood management practices from static approaches to an optimization-based real-time control (RTC) of urban drainage systems. Recent studies have addressed a plethora of strategies for flood control in stormwater reservoirs; however, advanced control theoretic techniques are not yet fully investigated and applied to these systems. In addition, there is an absence of a coupled integrated control model for systems composed of watersheds, reservoirs, and channels for flood mitigation. To this end, we develop a novel state-space model of hydrologic and hydrodynamic processes in reservoirs and one-dimensional channels. The model is tested under different types of reservoir control strategies based on real-time measurements (reactive control), and based on predictions of the future behavior of the system (predictive control) using rainfall forecastings. We apply the modeling approach in a system composed by a single watershed, reservoir, and a channel connected in series, respectively, for the San Antonio observed rainfall data. Results indicate that for flood mitigation, the predictive control strategy outperforms the reactive controls not only when applied for synthetic design storm events, but also for a continuous simulation. Moreover, the predictive control strategy requires smaller valve operations, while still guaranteeing efficient hydrological performance. From the results, we recommend the use of the model predictive control strategy to control stormwater systems due to the ability to handle different objective functions, which can be altered according to rainfall forecasting and shift the reservoir operation from flood-based control to strategies focused on increasing detention times, depending on the forecasting.
Modern control systems must operate in increasingly complex environments subject to safety constraints and input limits, and are often implemented in a hierarchical fashion with different controllers running at multiple time scales. Yet traditional constructive methods for nonlinear controller synthesis typically "flatten" this hierarchy, focusing on a single time scale, and thereby limited the ability to make rigorous guarantees on constraint satisfaction that hold for the entire system. In this work we seek to address the stabilization of constrained nonlinear systems through a \textit{multi-rate} control architecture. This is accomplished by iteratively planning continuous reference trajectories for a nonlinear system using a linearized model and Model Predictive Control (MPC), and tracking said trajectories using the full-order nonlinear model and Control Lyapunov Functions (CLFs). Connecting these two levels of control design in a way that ensures constraint satisfaction is achieved through the use of \textit{B\'{e}zier curves}, which enable planning continuous trajectories respecting constraints by planning a sequence of discrete points. Our framework is encoded via convex optimization problems which may be efficiently solved, as demonstrated in simulation.
We propose a robust model predictive control (MPC) method for discrete-time linear systems with polytopic model uncertainty and additive disturbances. Optimizing over linear time-varying (LTV) state feedback controllers has been successfully used for robust MPC when only additive disturbances are present. However, it is challenging to design LTV state feedback controllers in the face of model uncertainty whose effects are difficult to bound. To address this issue, we propose a novel approach to over-approximate the effects of both model uncertainty and additive disturbances by a filtered additive disturbance signal. Using the System Level Synthesis framework, we jointly search for robust LTV state feedback controllers and the bounds on the effects of uncertainty online, which allows us to reduce the conservatism and minimize an upper bound on the worst-case cost in robust MPC. We provide a comprehensive numerical comparison of our method and representative robust MPC methods from the literature. Numerical examples demonstrate that our proposed method can significantly reduce the conservatism over a wide range of uncertainty parameters with comparable computational effort as the baseline methods.
Model Predictive Control (MPC) has become a popular framework in embedded control for high-performance autonomous systems. However, to achieve good control performance using MPC, an accurate dynamics model is key. To maintain real-time operation, the dynamics models used on embedded systems have been limited to simple first-principle models, which substantially limits their representative power. In contrast to such simple models, machine learning approaches, specifically neural networks, have been shown to accurately model even complex dynamic effects, but their large computational complexity hindered combination with fast real-time iteration loops. With this work, we present Real-time Neural MPC, a framework to efficiently integrate large, complex neural network architectures as dynamics models within a model-predictive control pipeline. Our experiments, performed in simulation and the real world onboard a highly agile quadrotor platform, demonstrate the capabilities of the described system to run learned models with, previously infeasible, large modeling capacity using gradient-based online optimization MPC. Compared to prior implementations of neural networks in online optimization MPC we can leverage models of over 4000 times larger parametric capacity in a 50Hz real-time window on an embedded platform. Further, we show the feasibility of our framework on real-world problems by reducing the positional tracking error by up to 82% when compared to state-of-the-art MPC approaches without neural network dynamics.
Data-driven model predictive control has two key advantages over model-free methods: a potential for improved sample efficiency through model learning, and better performance as computational budget for planning increases. However, it is both costly to plan over long horizons and challenging to obtain an accurate model of the environment. In this work, we combine the strengths of model-free and model-based methods. We use a learned task-oriented latent dynamics model for local trajectory optimization over a short horizon, and use a learned terminal value function to estimate long-term return, both of which are learned jointly by temporal difference learning. Our method, TD-MPC, achieves superior sample efficiency and asymptotic performance over prior work on both state and image-based continuous control tasks from DMControl and Meta-World. Code and video results are available at https://nicklashansen.github.io/td-mpc.
The problem of synthesizing stochastic explicit model predictive control policies is known to be quickly intractable even for systems of modest complexity when using classical control-theoretic methods. To address this challenge, we present a scalable alternative called stochastic parametric differentiable predictive control (SP-DPC) for unsupervised learning of neural control policies governing stochastic linear systems subject to nonlinear chance constraints. SP-DPC is formulated as a deterministic approximation to the stochastic parametric constrained optimal control problem. This formulation allows us to directly compute the policy gradients via automatic differentiation of the problem's value function, evaluated over sampled parameters and uncertainties. In particular, the computed expectation of the SP-DPC problem's value function is backpropagated through the closed-loop system rollouts parametrized by a known nominal system dynamics model and neural control policy which allows for direct model-based policy optimization. We provide theoretical probabilistic guarantees for policies learned via the SP-DPC method on closed-loop stability and chance constraints satisfaction. Furthermore, we demonstrate the computational efficiency and scalability of the proposed policy optimization algorithm in three numerical examples, including systems with a large number of states or subject to nonlinear constraints.
The ability to accurately predict human behavior is central to the safety and efficiency of robot autonomy in interactive settings. Unfortunately, robots often lack access to key information on which these predictions may hinge, such as people's goals, attention, and willingness to cooperate. Dual control theory addresses this challenge by treating unknown parameters of a predictive model as stochastic hidden states and inferring their values at runtime using information gathered during system operation. While able to optimally and automatically trade off exploration and exploitation, dual control is computationally intractable for general interactive motion planning, mainly due to the fundamental coupling between robot trajectory optimization and human intent inference. In this paper, we present a novel algorithmic approach to enable active uncertainty reduction for interactive motion planning based on the implicit dual control paradigm. Our approach relies on sampling-based approximation of stochastic dynamic programming, leading to a model predictive control problem that can be readily solved by real-time gradient-based optimization methods. The resulting policy is shown to preserve the dual control effect for a broad class of predictive human models with both continuous and categorical uncertainty. The efficacy of our approach is demonstrated with simulated driving examples.
Reinforcement learning (RL) is a promising, upcoming topic in automatic control applications. Where classical control approaches require a priori system knowledge, data-driven control approaches like RL allow a model-free controller design procedure, rendering them emergent techniques for systems with changing plant structures and varying parameters. While it was already shown in various applications that the transient control behavior for complex systems can be sufficiently handled by RL, the challenge of non-vanishing steady-state control errors remains, which arises from the usage of control policy approximations and finite training times. To overcome this issue, an integral action state augmentation (IASA) for actor-critic-based RL controllers is introduced that mimics an integrating feedback, which is inspired by the delta-input formulation within model predictive control. This augmentation does not require any expert knowledge, leaving the approach model free. As a result, the RL controller learns how to suppress steady-state control deviations much more effectively. Two exemplary applications from the domain of electrical energy engineering validate the benefit of the developed method both for reference tracking and disturbance rejection. In comparison to a standard deep deterministic policy gradient (DDPG) setup, the suggested IASA extension allows to reduce the steady-state error by up to 52 $\%$ within the considered validation scenarios.
Insect flight is a strongly nonlinear and actuated dynamical system. As such, strategies for understanding its control have typically relied on either model-based methods or linearizations thereof. Here we develop a framework that combines model predictive control on an established flight dynamics model and deep neural networks (DNN) to create an efficient method for solving the inverse problem of flight control. We turn to natural systems for inspiration since they inherently demonstrate network pruning with the consequence of yielding more efficient networks for a specific set of tasks. This bio-inspired approach allows us to leverage network pruning to optimally sparsify a DNN architecture in order to perform flight tasks with as few neural connections as possible, however, there are limits to sparsification. Specifically, as the number of connections falls below a critical threshold, flight performance drops considerably. We develop sparsification paradigms and explore their limits for control tasks. Monte Carlo simulations also quantify the statistical distribution of network weights during pruning given initial random weights of the DNNs. We demonstrate that on average, the network can be pruned to retain approximately 7% of the original network weights, with statistical distributions quantified at each layer of the network. Overall, this work shows that sparsely connected DNNs are capable of predicting the forces required to follow flight trajectories. Additionally, sparsification has sharp performance limits.
Robust autonomy stacks require tight integration of perception, motion planning, and control layers, but these layers often inadequately incorporate inherent perception and prediction uncertainties, either ignoring them altogether or making questionable assumptions of Gaussianity. Robots with nonlinear dynamics and complex sensing modalities operating in an uncertain environment demand more careful consideration of how uncertainties propagate across stack layers. We propose a framework to integrate perception, motion planning, and control by explicitly incorporating perception and prediction uncertainties into planning so that risks of constraint violation can be mitigated. Specifically, we use a nonlinear model predictive control based steering law coupled with a decorrelation scheme based Unscented Kalman Filter for state and environment estimation to propagate the robot state and environment uncertainties. Subsequently, we use distributionally robust risk constraints to limit the risk in the presence of these uncertainties. Finally, we present a layered autonomy stack consisting of a nonlinear steering-based distributionally robust motion planning module and a reference trajectory tracking module. Our numerical experiments with nonlinear robot models and an urban driving simulator show the effectiveness of our proposed approaches.
This paper proposes a deep learning-based optimal battery management scheme for frequency regulation (FR) by integrating model predictive control (MPC), supervised learning (SL), reinforcement learning (RL), and high-fidelity battery models. By taking advantage of deep neural networks (DNNs), the derived DNN-approximated policy is computationally efficient in online implementation. The design procedure of the proposed scheme consists of two sequential processes: (1) the SL process, in which we first run a simulation with an MPC embedding a low-fidelity battery model to generate a training data set, and then, based on the generated data set, we optimize a DNN-approximated policy using SL algorithms; and (2) the RL process, in which we utilize RL algorithms to improve the performance of the DNN-approximated policy by balancing short-term economic incentives and long-term battery degradation. The SL process speeds up the subsequent RL process by providing a good initialization. By utilizing RL algorithms, one prominent property of the proposed scheme is that it can learn from the data generated by simulating the FR policy on the high-fidelity battery simulator to adjust the DNN-approximated policy, which is originally based on low-fidelity battery model. A case study using real-world data of FR signals and prices is performed. Simulation results show that, compared to conventional MPC schemes, the proposed deep learning-based scheme can effectively achieve higher economic benefits of FR participation while maintaining lower online computational cost.
Motivated by large-scale but computationally constrained settings, e.g., the Internet of Things, we present a novel data-driven distributed control algorithm that is synthesized directly from trajectory data. Our method, data-driven Distributed and Localized Model Predictive Control (D$^3$LMPC), builds upon the data-driven System Level Synthesis (SLS) framework, which allows one to parameterize \emph{closed-loop} system responses directly from collected open-loop trajectories. The resulting model-predictive controller can be implemented with distributed computation and only local information sharing. By imposing locality constraints on the system response, we show that the amount of data needed for our synthesis problem is independent of the size of the global system. Moreover, we show that our algorithm enjoys theoretical guarantees for recursive feasibility and asymptotic stability. Finally, we also demonstrate the optimality and scalability of our algorithm in a simulation experiment.
We propose a robust model predictive control (MPC) method for discrete-time linear time-invariant systems with norm-bounded additive disturbances and model uncertainty. In our method, at each time step we solve a finite time robust optimal control problem (OCP) which jointly searches over robust linear state feedback controllers and bounds the deviation of the system states from the nominal predicted trajectory. By leveraging the System Level Synthesis (SLS) framework, the proposed robust OCP is formulated as a convex quadratic program in the space of closed-loop system responses. When an adaptive horizon strategy is used, we prove the recursive feasibility of the proposed MPC controller and input-to-state stability of the origin for the closed-loop system. We demonstrate through numerical examples that the proposed method considerably reduces conservatism when compared with existing SLS-based and tube-based robust MPC methods, while also enjoying low computational complexity.
Model predictive control (MPC) has been used widely in power electronics due to its simple concept, fast dynamic response, and good reference tracking. However, it suffers from parametric uncertainties, since it directly relies on the mathematical model of the system to predict the optimal switching states to be used at the next sampling time. As a result, uncertain parameters lead to an ill-designed MPC. Thus, this paper offers a model-free control strategy on the basis of artificial neural networks (ANNs), for mitigating the effects of parameter mismatching while having a little negative impact on the inverter's performance. This method includes two related stages. First, MPC is used as an expert to control the studied converter in order to provide a dataset, while, in the second stage, the obtained dataset is utilized to train the proposed ANN. The case study herein is based on a four-level three-cell flying capacitor inverter. In this study, MATLAB/Simulink is used to simulate the performance of the proposed method, taking into account various operating conditions. Afterward, the simulation results are reported in comparison with the conventional MPC scheme, demonstrating the superior performance of the proposed control strategy in terms of robustness against parameters mismatch and low total harmonic distortion (THD), especially when changes occur in the system parameters, compared to the conventional MPC. Furthermore, the experimental validation of the proposed method is provided based on the Hardware-in-the-Loop (HIL) simulation using the C2000TM-microcontroller-LaunchPadXL TMS320F28379D kit, demonstrating the applicability of the ANN-based control strategy to be implemented on a DSP controller.
This tutorial shows an overview of Model Predictive Control with a linear discrete-time system and constrained states and inputs. The focus is on the implementation of the method under consideration of stability and recursive feasibility. The MATLAB code for the examples and plots is available online.