In this paper, we are interested in optimal control problems with purely economic costs, which often yield optimal policies having a (nearly) bang-bang structure. We focus on policy approximations based on Model Predictive Control (MPC) and the use of the deterministic policy gradient method to optimize the MPC closed-loop performance in the presence of unmodelled stochasticity or model error. When the policy has a (nearly) bang-bang structure, we observe that the policy gradient method can struggle to produce meaningful steps in the policy parameters. To tackle this issue, we propose a homotopy strategy based on the interior-point method, providing a relaxation of the policy during the learning. We investigate a specific well-known battery storage problem, and show that the proposed method delivers a homogeneous and faster learning than a classical policy gradient approach.
As robotic systems move from highly structured environments to open worlds, incorporating uncertainty from dynamics learning or state estimation into the control pipeline is essential for robust performance. In this paper we present a nonlinear particle model predictive control (PMPC) approach to control under uncertainty, which directly incorporates any particle-based uncertainty representation, such as those common in robotics. Our approach builds on scenario methods for MPC, but in contrast to existing approaches, which either constrain all or only the first timestep to share actions across scenarios, we investigate the impact of a \textit{partial consensus horizon}. Implementing this optimization for nonlinear dynamics by leveraging sequential convex optimization, our approach yields an efficient framework that can be tuned to the particular information gain dynamics of a system to mitigate both over-conservatism and over-optimism. We investigate our approach for two robotic systems across three problem settings: time-varying, partially observed dynamics; sensing uncertainty; and model-based reinforcement learning, and show that our approach improves performance over baselines in all settings.
Accounting for more than 40% of global energy consumption, residential and commercial buildings will be key players in any future green energy systems. To fully exploit their potential while ensuring occupant comfort, a robust control scheme is required to handle various uncertainties, such as external weather and occupant behaviour. However, prominent patterns, especially periodicity, are widely seen in most sources of uncertainty. This paper incorporates this correlated structure into the learning model predictive control framework, in order to learn a global optimal robust control scheme for building operations.
Physics-informed neural networks (PINNs) impose known physical laws into the learning of deep neural networks, making sure they respect the physics of the process while decreasing the demand of labeled data. For systems represented by Ordinary Differential Equations (ODEs), the conventional PINN has a continuous time input variable and outputs the solution of the corresponding ODE. In their original form, PINNs do not allow control inputs, neither can they simulate for variable long-range intervals without serious degradation in their predictions. In this context, this work presents a new framework called Physics-Informed Neural Nets for Control (PINC), which proposes a novel PINN-based architecture that is amenable to control problems and able to simulate for longer-range time horizons that are not fixed beforehand, making it a very flexible framework when compared to traditional PINNs. Furthermore, this long-range time simulation of differential equations is faster than numerical methods since it relies only on signal propagation through the network, making it less computationally costly and, thus, a better alternative for simulation of models in Model Predictive Control. We showcase our proposal in the control of two nonlinear dynamic systems: the Van der Pol oscillator and the four-tank system.
In this work, we propose a novel missile guidance algorithm that combines deep learning based trajectory prediction with nonlinear model predictive control. Although missile guidance and threat interception is a well-studied problem, existing algorithms' performance degrades significantly when the target is pulling high acceleration attack maneuvers while rapidly changing its direction. We argue that since most threats execute similar attack maneuvers, these nonlinear trajectory patterns can be processed with modern machine learning methods to build high accuracy trajectory prediction algorithms. We train a long short-term memory network (LSTM) based on a class of simulated structured agile attack patterns, then combine this predictor with quadratic programming based nonlinear model predictive control (NMPC). Our method, named nonlinear model based predictive control with target acceleration predictions (NMPC-TAP), significantly outperforms compared approaches in terms of miss distance, for the scenarios where the target/threat is executing agile maneuvers.
Control co-design (CCD) is a technique for improving the closed-loop performance of systems through the coordinated design of both plant parameters and an optimal control policy. While model predictive control (MPC) is an attractive control strategy for many systems, embedding it within a CCD algorithm presents challenges because obtaining a closed-form solution for this receding-horizon optimization strategy is often not feasible. This paper meets that challenge by including a robust MPC formulation within the inner loop of a CCD algorithm. As exemplified by application to an aircraft thermal management system, the proposed algorithm closely matches the plant design of an open-loop benchmark. However, unlike the open-loop approach, the proposed algorithm can leverage MPC control variables designed a priori to achieve robust online operation under disturbance profiles that differ from those used for design.
The application of multilevel converters to renewable energy systems is a growing topic due to their advantages in energy efficiency. Regarding its control, model predictive control (MPC) has become very appealing due to its natural consideration of discrete inputs, its optimization capability, and the present-day availability of powerful processing hardware. The main drawback of MPC compared to other control techniques in this field is that the control input is held constant during the sampling period, and it is usually difficult or even impossible to reduce this sampling period because of hardware limitations. For this reason, a multirate MPC algorithm is proposed, which allows to change the control input several times within the sampling period. The optimization problem is simplified and made suboptimal to substantially decrease computational burden. This approach is tested in simulation on a three-phase, five-level diode-clamped converter (DCC) operating in inverted mode with a three-phase resistive load. Results show significant reduction in harmonic distortion at the cost of an increase in the number of commutations with respect to a standard MPC operating at the same sampling period.
One of the most critical tasks in tractor operation is the accurate steering during field operations, e.g., accurate trajectory following during mechanical weeding or spraying, to avoid damaging the crop or planting when there is no crop yet. To automate the trajectory following problem of an autonomous tractor-trailer system and also increase its steering accuracy, a nonlinear model predictive control approach has been proposed in this paper. For the state and parameter estimation, moving horizon estimation has been chosen since it considers the state and the parameter estimation within the same problem and also constraints both on inputs and states can be incorporated. The experimental results show the accuracy and the efficiency of the proposed control scheme in which the mean values of the Euclidean error for the tractor and the trailer, respectively, are 6.44 and 3.61 cm for a straight line trajectory and 49.78 and 41.52 cm for a curved line trajectory.
Advanced building control methods such as model predictive control (MPC) offer significant potential benefits to both consumers and grid operators, but the high computational requirements have acted as barriers to more widespread adoption. Local control computation requires installation of expensive computational hardware, while cloud computing introduces data security and privacy concerns. In this paper, we drastically reduce the local computational requirements of advanced building control through a reinforcement learning (RL)-based approach called Behavioral Cloning, which represents the MPC policy as a neural network that can be locally implemented and quickly computed on a low-cost programmable logic controller. While previous RL and approximate MPC methods must be specifically trained for each building, our key improvement is that our controller can generalize to many buildings, electricity rates, and thermostat setpoint schedules without additional, effort-intensive retraining. To provide this versatility, we have adapted the traditional Behavioral Cloning approach through (1) a constraint-informed parameter grouping (CIPG) method that provides a more efficient representation of the training data; (2) an MPC-Guided training data generation method using the DAgger algorithm that improves stability and constraint satisfaction; and (3) a new deep learning model-structure called reverse-time recurrent neural networks (RT-RNN) that allows future information to flow backward in time to more effectively interpret the temporal information in disturbance predictions. The result is an easy-to-deploy, generalized behavioral clone of MPC that can be implemented on a programmable logic controller and requires little building-specific controller tuning, reducing the effort and costs associated with implementing smart residential heat pump control.
In this paper, we provide a generalized framework for Variational Inference-Stochastic Optimal Control by using thenon-extensive Tsallis divergence. By incorporating the deformed exponential function into the optimality likelihood function, a novel Tsallis Variational Inference-Model Predictive Control algorithm is derived, which includes prior works such as Variational Inference-Model Predictive Control, Model Predictive PathIntegral Control, Cross Entropy Method, and Stein VariationalInference Model Predictive Control as special cases. The proposed algorithm allows for effective control of the cost/reward transform and is characterized by superior performance in terms of mean and variance reduction of the associated cost. The aforementioned features are supported by a theoretical and numerical analysis on the level of risk sensitivity of the proposed algorithm as well as simulation experiments on 5 different robotic systems with 3 different policy parameterizations.
We are motivated by the goal of generalist robots that can complete a wide range of tasks across many environments. Critical to this is the robot's ability to acquire some metric of task success or reward, which is necessary for reinforcement learning, planning, or knowing when to ask for help. For a general-purpose robot operating in the real world, this reward function must also be able to generalize broadly across environments, tasks, and objects, while depending only on on-board sensor observations (e.g. RGB images). While deep learning on large and diverse datasets has shown promise as a path towards such generalization in computer vision and natural language, collecting high quality datasets of robotic interaction at scale remains an open challenge. In contrast, "in-the-wild" videos of humans (e.g. YouTube) contain an extensive collection of people doing interesting tasks across a diverse range of settings. In this work, we propose a simple approach, Domain-agnostic Video Discriminator (DVD), that learns multitask reward functions by training a discriminator to classify whether two videos are performing the same task, and can generalize by virtue of learning from a small amount of robot data with a broad dataset of human videos. We find that by leveraging diverse human datasets, this reward function (a) can generalize zero shot to unseen environments, (b) generalize zero shot to unseen tasks, and (c) can be combined with visual model predictive control to solve robotic manipulation tasks on a real WidowX200 robot in an unseen environment from a single human demo.
This paper is concerned with the design of intermittent non-pharmaceutical strategies to mitigate the spread of the COVID-19 epidemic exploiting network epidemiological models. Specifically, by studying a variational equation for the dynamics of the infected in a network model of the epidemic spread, we derive, using contractivity arguments, a condition that can be used to guarantee that, in epidemiological terms, the effective reproduction number is less than unity. This condition has three advantages: (i) it is easily computable; (ii) it is directly related to the model parameters; (iii) it can be used to enforce a scalability condition that prohibits the amplification of disturbances within the network system. We then include satisfaction of such a condition as a constraint in a Model Predictive Control problem so as to mitigate (or suppress) the spread of the epidemic while minimizing the economic impact of the interventions. A data-driven model of Italy as a network of three macro-regions (North, Center, and South), whose parameters are identified from real data, is used to illustrate and evaluate the effectiveness of the proposed control strategy.
We employ model predictive control for a multi-period portfolio optimization problem. In addition to the mean-variance objective, we construct a portfolio whose allocation is given by model predictive control with a risk-parity objective, and provide a successive convex program algorithm that provides 30 times faster and robust solutions in the experiments. Computational results on the multi-asset universe show that multi-period models perform better than their single period counterparts in out-of-sample period, 2006-2020. The out-of-sample risk-adjusted performance of both mean-variance and risk-parity formulations beat the fix-mix benchmark, and achieve Sharpe ratio of 0.64 and 0.97, respectively.
We investigate adaptive strategies to robustly and optimally control the COVID-19 pandemic via social distancing measures based on the example of Germany. Our goal is to minimize the number of fatalities over the course of two years without inducing excessive social costs. We consider a tailored model of the German COVID-19 outbreak with different parameter sets to design and validate our approach. Our analysis reveals that an open-loop optimal control policy can significantly decrease the number of fatalities when compared to simpler policies under the assumption of exact model knowledge. In a more realistic scenario with uncertain data and model mismatch, a feedback strategy that updates the policy weekly using model predictive control (MPC) leads to a reliable performance, even when applied to a validation model with deviant parameters. On top of that, we propose a robust MPC-based feedback policy using interval arithmetic that adapts the social distancing measures cautiously and safely, thus leading to a minimum number of fatalities even if measurements are inaccurate and the infection rates cannot be precisely specified by social distancing. Our theoretical findings support various recent studies by showing that 1) adaptive feedback strategies are required to reliably contain the COVID-19 outbreak, 2) well-designed policies can significantly reduce the number of fatalities compared to simpler ones while keeping the amount of social distancing measures on the same level, and 3) imposing stronger social distancing measures early on is more effective and cheaper in the long run than opening up too soon and restoring stricter measures at a later time.
This article proposes a new safety concept: backup plan safety. The backup plan safety is defined as the ability to complete one of the alternative missions in the case of primary mission abortion. To incorporate this new safety concept in control problems, we formulate a feasibility maximization problem that adopts additional (virtual) input horizons toward the alternative missions on top of the input horizon toward the primary mission. Cost functions for the primary and alternative missions construct multiple objectives, and multi-horizon inputs evaluate them. To address the feasibility maximization problem, we develop a multi-horizon multi-objective model predictive path integral control (3M) algorithm. Model predictive path integral control (MPPI) is a sampling-based scheme that can help the proposed algorithm deal with nonlinear dynamic systems and achieve computational efficiency by parallel computation. Simulations of the aerial vehicle and ground vehicle control problems demonstrate the new concept of backup plan safety and the performance of the proposed algorithm.
Learning-based model predictive control (MPC) is an approach designed to reduce the computational cost of MPC. In this paper, a constrained deep neural network (DNN) design is proposed to learn MPC policy for nonlinear systems. Using constrained training of neural networks, MPC constraints are enforced effectively. Furthermore, recursive feasibility and robust stability conditions are derived for the learning-based MPC approach. Additionally, probabilistic feasibility and optimality empirical guarantees are provided for the learned control policy. The proposed algorithm is implemented on the Furuta pendulum and control performance is demonstrated and compared with the exact MPC and the normally trained learning-based MPC. The results show superior control performance and constraint satisfaction of the proposed approach.
This thesis is concerned with the rejection of time-varying disturbances in linear model predictive control of discrete-time systems. In the literature, disturbances are widely rejected by using velocity models, disturbance model with observer approach or a scheme that combines the compensation of a disturbance observer and the feedback regulation of MPC. Contrary to the widely used methods, the technique proposed in this research utilises the increment model of plants, with the assumption of fast-changing disturbances, to formulate a control law to reject the varying-disturbances. The uniqueness of the method stems from the compensation of the disturbance magnitude and rate of change. By proposing a cost function where the increment form of the system disturbance is taken as an optimisation variable, a control signal that is a function of a computed optimal disturbance increment is formulated to ensure that the plant is driven according to the minimisation of the cost function. The degree of freedom introduced by using the optimal disturbance in the control law was exploited to introduce the estimated disturbance increment into the control signal such that it is always in opposition to the external disturbance increment. Moreover, the proposed cost function provides a weighting matrix that can be used to manipulate the impacts of the exogenous disturbances on the response of the system. To estimate the unmeasurable disturbances, a combined state and disturbance observer is designed based on a convex optimisation stated in terms of an H2-minimisation problem. Simulations of three different systems are used to show the benefits of the proposed algorithm when compared with conventional offset-free MPC techniques. The results demonstrated that the proposed scheme can give significantly improved output tracking and regulation.
Trajectory optimization and model predictive control are essential techniques underpinning advanced robotic applications, ranging from autonomous driving to full-body humanoid control. State-of-the-art algorithms have focused on data-driven approaches that infer the system dynamics online and incorporate posterior uncertainty during planning and control. Despite their success, such approaches are still susceptible to catastrophic errors that may arise due to statistical learning biases, unmodeled disturbances, or even directed adversarial attacks. In this paper, we tackle the problem of dynamics mismatch and propose a distributionally robust optimal control formulation that alternates between two relative entropy trust-region optimization problems. Our method finds the worst-case maximum entropy Gaussian posterior over the dynamics parameters and the corresponding robust policy. Furthermore, we show that our approach admits a closed-form backward-pass for a certain class of systems. Finally, we demonstrate the resulting robustness on linear and nonlinear numerical examples.
Electrical neuromodulation as a palliative treatment has been increasingly used in the control of epilepsy. However, current neuromodulations commonly implement predetermined actuation strategies and lack the capability of self-adaptively adjusting stimulation inputs. In this work, rooted in optimal control theory, we propose a Koopman-MPC framework for real-time closed-loop electrical neuromodulation in epilepsy, which integrates i) a deep Koopman operator based dynamical model to predict the temporal evolution of epileptic EEG with an approximate finite-dimensional linear dynamics and ii) a model predictive control (MPC) module to design optimal seizure suppression strategies. The Koopman operator based linear dynamical model is embedded in the latent state space of the autoencoder neural network, in which we can approximate and update the Koopman operator online. The linear dynamical property of the Koopman operator ensures the convexity of the optimization problem for subsequent MPC control. The proposed deep Koopman operator model shows greater predictive capability than the baseline models (e.g., vector autoregressive model, kernel based method and recurrent neural network (RNN)) in both synthetic and real epileptic EEG data. Moreover, compared with the RNN-MPC framework, our Koopman-MPC framework can suppress seizure dynamics with better computational efficiency in both the Jansen-Rit model and the Epileptor model. Koopman-MPC framework opens a new window for model-based closed-loop neuromodulation and sheds light on nonlinear neurodynamics and feedback control policies.