Model-predictive-control (MPC) offers an optimal control technique to establish and ensure that the total operation cost of multi-energy systems remains at a minimum while fulfilling all system constraints. However, this method presumes an adequate model of the underlying system dynamics, which is prone to modelling errors and is not necessarily adaptive. This has an associated initial and ongoing project-specific engineering cost. In this paper, we present an on- and off-policy multi-objective reinforcement learning (RL) approach, that does not assume a model a priori, benchmarking this against a linear MPC (LMPC - to reflect current practice, though non-linear MPC performs better) - both derived from the general optimal control problem, highlighting their differences and similarities. In a simple multi-energy system (MES) configuration case study, we show that a twin delayed deep deterministic policy gradient (TD3) RL agent offers potential to match and outperform the perfect foresight LMPC benchmark (101.5%). This while the realistic LMPC, i.e. imperfect predictions, only achieves 98%. While in a more complex MES system configuration, the RL agent's performance is generally lower (94.6%), yet still better than the realistic LMPC (88.9%). In both case studies, the RL agents outperformed the realistic LMPC after a training period of 2 years using quarterly interactions with the environment. We conclude that reinforcement learning is a viable optimal control technique for multi-energy systems given adequate constraint handling and pre-training, to avoid unsafe interactions and long training periods, as is proposed in fundamental future work.
This paper proposes an ecological adaptive cruise control (EACC) concept with the primary goal to minimize the fuel consumption in a city bus with an internal combustion engine (ICE). A hybrid model predictive control (HMPC) is implemented in this work to control both continuous and discrete-time variables. Moreover, a multi-objective optimization problem for EACC is formulated in time-domain as a mixed-integer quadratically constrained quadratic programming (MIQCQP) problem. The proposed HMPC-EACC performs robust vehicle-following while tracking a leading vehicle and plans fuel-efficient acceleration and deceleration maneuvers for the host vehicle. Additionally, it uses the signal phase and timing (SPaT) information to compute a green wave reference speed for the host vehicle to cross the signalized intersections at a green phase. Moreover, the proposed controller performs pulse and glide (PnG) to optimally control the engine ON and OFF states and save additional fuel. Furthermore, the performance of the proposed strategy is evaluated on a real-world driving profile and compared against a baseline controller from the literature. Finally, the influence of different prediction horizons on the fuel savings and computation times are studied. The results reveal significant reduction in fuel consumption with HMPC-EACC and demonstrate that the proposed controller is real-time capable.
We present a simplified model of a vehicle driving on a nonplanar road. A parametric surface is used to describe the nonplanar road which can describe any combination of curvature, bank and slope. We show that the proposed modeling approach generalizes planar vehicle models that reference a centerline, such as the Frenet model. We use the proposed approach for vehicle path planning and following using model predictive control. We also model and control vehicle contact with the road surface. We demonstrate that the proposed controller improves speed and lane following on complex roads compared to planar vehicle controllers, and mitigates loss of control on complex road surfaces including off-camber turns.
We propose a robust data-driven model predictive control (MPC) scheme to control linear time-invariant (LTI) systems. The scheme uses an implicit model description based on behavioral systems theory and past measured trajectories. In particular, it does not require any prior identification step, but only an initially measured input-output trajectory as well as an upper bound on the order of the unknown system. First, we prove exponential stability of a nominal data-driven MPC scheme with terminal equality constraints in the case of no measurement noise. For bounded additive output measurement noise, we propose a robust modification of the scheme, including a slack variable with regularization in the cost. We prove that the application of this robust MPC scheme in a multi-step fashion leads to practical exponential stability of the closed loop w.r.t. the noise level. The presented results provide the first (theoretical) analysis of closed-loop properties, resulting from a simple, purely data-driven MPC scheme.
In this paper, we present an iterative Model Predictive Control (MPC) design for piecewise nonlinear systems. We consider finite time control tasks where the goal of the controller is to steer the system from a starting configuration to a goal state while minimizing a cost function. First, we present an algorithm that leverages a feasible trajectory that completes the task to construct a control policy which guarantees that state and input constraints are recursively satisfied and that the closed-loop system reaches the goal state in finite time. Utilizing this construction, we present a policy iteration scheme that iteratively generates safe trajectories which have non-decreasing performance. Finally, we test the proposed strategy on a discretized Spring Loaded Inverted Pendulum (SLIP) model with massless legs. We show that our methodology is robust to changes in initial conditions and disturbances acting on the system. Furthermore, we demonstrate the effectiveness of our policy iteration algorithm in a minimum time control task.
We propose a learning-based robust predictive control algorithm that compensates for significant uncertainty in the dynamics for a class of discrete-time systems that are nominally linear with an additive nonlinear component. Such systems commonly model the nonlinear effects of an unknown environment on a nominal system. We optimize over a class of nonlinear feedback policies inspired by certainty equivalent "estimate-and-cancel" control laws pioneered in classical adaptive control to achieve significant performance improvements in the presence of uncertainties of large magnitude, a setting in which existing learning-based predictive control algorithms often struggle to guarantee safety. In contrast to previous work in robust adaptive MPC, our approach allows us to take advantage of structure (i.e., the numerical predictions) in the a priori unknown dynamics learned online through function approximation. Our approach also extends typical nonlinear adaptive control methods to systems with state and input constraints even when we cannot directly cancel the additive uncertain function from the dynamics. Moreover, we apply contemporary statistical estimation techniques to certify the system's safety through persistent constraint satisfaction with high probability. Finally, we show in simulation that our method can accommodate more significant unknown dynamics terms than existing methods.
This paper presents a stability analysis tool for model predictive control (MPC) where control action is generated by optimising a cost function over a finite horizon. Stability analysis of MPC with a limited horizon but without terminal weight is a well known challenging problem. We define a new value function based on an auxiliary one-step optimisation related to stage cost, namely optimal one-step value function (OSVF). It is shown that a finite horizon MPC can be made to be asymptotically stable if OSVF is a (local) control Lyapunov function (CLF). More specifically, by exploiting the CLF property of OSFV to construct a contractive terminal set, a new stabilising MPC algorithm (CMPC) is proposed. We show that CMPC is recursively feasible and guarantees stability under the condition that OSVF is a CLF. Checking this condition and estimation of the maximal terminal set are discussed. Numerical examples are presented to demonstrate the effectiveness of the proposed stability condition and corresponding CMPC algorithm.
More efficient agricultural machinery is needed as agricultural areas become more limited and energy and labor costs increase. To increase their efficiency, trajectory tracking problem of an autonomous tractor, as an agricultural production machine, has been investigated in this study. As a widely used model-based approach, model predictive control is preferred in this paper to control the yaw dynamics of the tractor which can deal with the constraints on the states and the actuators in a system. The yaw dynamics is identified by using nonlinear least squares frequency domain system identification. The speed is controlled by a proportional-integral-derivative controller and a kinematic trajectory controller is used to calculate the desired speed and the desired yaw rate signals for the subsystems in order to minimize the tracking errors in both the longitudinal and transversal directions. The experimental results show the accuracy and the efficiency of the proposed control scheme in which the euclidean error is below $40$ cm for time-based straight line trajectories and $60$ cm for time-based curved line trajectories, respectively.
A path-planning algorithm for connected and non-connected automated road vehicles on multilane motorways is derived from the opportune formulation of an optimal control problem. In this framework, the objective function to be minimized contains appropriate respective terms to reflect: the goals of vehicle advancement; passenger comfort; and avoidance of collisions with other vehicles, of road departures and of negative speeds. Connectivity implies that connected vehicles are able to exchange with each other (V2V) or the infrastructure (V2I), real-time information about their last generated path. For the numerical solution of the optimal control problem, an efficient feasible direction algorithm is used. To ensure high-quality local minima, a simplified Dynamic Programming algorithm is also conceived to deliver the initial guess trajectory for the feasible direction algorithm. Thanks to low computation times, the approach is readily executable within a model predictive control (MPC) framework. The proposed MPC-based approach is embedded within the Aimsun microsimulation platform, which enables the evaluation of a plethora of realistic vehicle driving and advancement scenarios. Results obtained on a multilane motorway stretch indicate higher efficiency of the optimally controlled vehicles in driving closer to their desired speed, compared to ordinary Aimsun vehicles. Increased penetration rates of automated vehicles are found to increase the efficiency of the overall traffic flow, benefiting manual vehicles as well. Moreover, connected controlled vehicles appear to be more efficient compared to the corresponding non-connected controlled vehicles, due to the improved real-time information and short-term prediction.
Many popular approaches in the field of robust model predictive control (MPC) are based on nominal predictions. This paper presents a novel formulation of this class of controller with proven input-to-state stability and robust constraint satisfaction. Its advantages are: (i) the design of its main ingredients are tractable for medium to large-sized systems, (ii) the terminal set does not need to be robust with respect to all the possible system uncertainties, but only for a reduced set that can be made arbitrarily small, thus facilitating its design and implementation, (iii) under certain conditions the terminal set can be taken as a positive invariant set of the nominal system, allowing us to use a terminal equality constraint, which facilitates its application to large-scale systems, and (iv) the complexity of its optimization problem is comparable to the non-robust MPC variant. We show numerical closed-loop results of its application to a multivariable chemical plant and compare it against other robust MPC formulations.
This paper presents a data-driven approach to the design of predictive controllers. The prediction matrices utilized in standard model predictive control (MPC) algorithms are typically constructed using knowledge of a system model such as, state-space or input-output models. Instead, we directly estimate the prediction matrices relating future outputs with current and future inputs from measured data, off-line. On-line, the developed data--driven predictive controller reduces to solving a quadratic program with a similar structure and complexity as linear MPC. Additionally, we develop a new procedure for estimating prediction matrices from data for predictive controllers with integral action, corresponding to the rate-based formulation of linear MPC. The effectiveness of the developed data-driven predictive controller is illustrated on position control of a linear motor model.
Battery energy storage systems (ESS) are widely used in microgrids to complement high renewables. However, the real-time energy management of microgrids with battery ESS is challenging in two aspects: 1) the evolution process of battery energy level is across-time coupled; 2) uncertainties unavoidably arise in the forecasting process for renewable generation. In this paper, a tube-based model predictive control (MPC) approach is innovatively proposed in accommodating the real-time energy management of microgrids with battery ESS. Firstly, a real-time operation model of battery, including the degradation cost and time-aware SoC range, is proposed for the battery ESS. In particular, the battery feature shallower-cheaper is depicted and the terminal SoC requirement is achieved. Secondly, two cascaded MPC controllers are designed in the proposed tube-based MPC, in which reference trajectories are generated by the nominal MPC without uncertainties, and then the ancillary MPC steers the actual trajectories to the nominal ones upon the realization of uncertainties. Specifically, in this paper, the battery SoC is viewed as the state variable of the system, while the generator power output and exchange power with the utility are seen as control variables. Lastly, numerous case studies demonstrate the effectiveness of the proposed approach, including both the low and high penetration level of renewables. Additional Monte Carlo simulations of consecutive 365 days show that the competitive ratio of the proposed approach is excellently below 1.10.
Willems' fundamental lemma asserts that all trajectories of a linear time-invariant system can be obtained from a finite number of measured ones, assuming that controllability and a persistency of excitation condition hold. We show that these two conditions can be relaxed. First, we prove that the controllability condition can be replaced by a condition on the controllable subspace, unobservable subspace, and a certain subspace associated with the measured trajectories. Second, we prove that the persistency of excitation requirement can be relaxed if the degree of a certain minimal polynomial is tightly bounded. Our results show that data-driven predictive control using online data is equivalent to model predictive control, even for uncontrollable systems. Moreover, our results significantly reduce the amount of data needed in identifying homogeneous multi-agent systems.
We consider the problem of risk-sensitive motion planning in the presence of randomly moving obstacles. To this end, we adopt a model predictive control (MPC) scheme and pose the obstacle avoidance constraint in the MPC problem as a distributionally robust constraint with a KL divergence ambiguity set. This constraint is the dual representation of the Entropic Value-at-Risk (EVaR). Building upon this viewpoint, we propose an algorithm to follow waypoints and discuss its feasibility and completion in finite time. We compare the policies obtained using EVaR with those obtained using another common coherent risk measure, Conditional Value-at-Risk (CVaR), via numerical experiments for a 2D system. We also implement the waypoint following algorithm on a 3D quadcopter simulation.
Individual Pitch Control (IPC) is an effective control strategy to mitigate the blade loads on large-scale wind turbines. Since IPC usually requires high pitch actuation, the safety constraints of the pitch actuator should be taken into account when designing the controller. This paper introduces a constrained Subspace Predictive Repetitive Control (SPRC) approach, which considers the limitation of blade pitch angle and pitch rate. To fulfill this goal, a model predictive control scheme is implemented in the fully data-driven SPRC approach to incorporate the physical limitations of the pitch actuator in the control problem formulation. An optimal control law subjected to constraints is then formulated so that future constraint violations are anticipated and prevented. Case studies show that the developed constrained SPRC reduces the pitch activities necessary to mitigate the blade loads when experiencing wind turbulence and abrupt wind gusts. More importantly, the approach allows the wind farm operator to design conservative bounds for the pitch actuator constraints that satisfies safety limitations, design specifications and physical restrictions. This will help to alleviate the cyclic fatigue loads on the actuators, increase the structural reliability and extend the lifespan of the pitch control system.
To improve the safety and efficiency of the intelligent transportation system, particularly in complex urban scenarios, in this paper a game theoretic decision-making framework is designed for connected automated vehicles (CAVs) at unsignalized roundabouts considering their personalized driving behaviours. Within the decision-making framework, a motion prediction module is designed and optimized using model predictive control (MPC) to enhance the effectiveness and accuracy of the decision-making algorithm. Besides, the payoff function of decision making is defined with the consideration of vehicle safety, ride comfort and travel efficiency. Additionally, the constraints of the decision-making problem are constructed. Based on the established decision-making model, Stackelberg game and grand coalition game approaches are adopted to address the decision making of CAVs at an unsignalized roundabout. Three testing cases considering personalized driving behaviours are carried out to verify the performance of the developed decision-making algorithms. The testing results show that the proposed game theoretic decision-making framework is able to make safe and reasonable decisions for CAVs in the complex urban scenarios, validating its feasibility and effectiveness. Stackelberg game approach shows its advantage in guaranteeing personalized driving objectives of individuals, while the grand coalition game approach is advantageous regarding the efficiency improvement of the transportation system.
This paper proposes a discretionary lane selection algorithm. In particular, highway driving is considered as a targeted scenario, where each lane has a different level of traffic flow. When lane-changing is discretionary, it is advised not to change lanes unless highly beneficial, e.g., reducing travel time significantly or securing higher safety. Evaluating such "benefit" is a challenge, along with multiple surrounding vehicles in dynamic speed and heading with uncertainty. We propose a real-time lane-selection algorithm with careful cost considerations and with modularity in design. The algorithm is a search-based optimization method that evaluates uncertain dynamic positions of other vehicles under a continuous time and space domain. For demonstration, we incorporate a state-of-the-art motion planner framework (Neural Networks integrated Model Predictive Control) under a CARLA simulation environment.
Model Predictive Control (MPC) is a classic tool for optimal control of complex, real-world systems. Although it has been successfully applied to a wide range of challenging tasks in robotics, it is fundamentally limited by the prediction horizon, which, if too short, will result in myopic decisions. Recently, several papers have suggested using a learned value function as the terminal cost for MPC. If the value function is accurate, it effectively allows MPC to reason over an infinite horizon. Unfortunately, Reinforcement Learning (RL) solutions to value function approximation can be difficult to realize for robotics tasks. In this paper, we suggest a more efficient method for value function approximation that applies to goal-directed problems, like reaching and navigation. In these problems, MPC is often formulated to track a path or trajectory returned by a planner. However, this strategy is brittle in that unexpected perturbations to the robot will require replanning, which can be costly at runtime. Instead, we show how the intermediate data structures used by modern planners can be interpreted as an approximate value function. We show that that this value function can be used by MPC directly, resulting in more efficient and resilient behavior at runtime.
We present a Reinforcement Learning-based Robust Nonlinear Model Predictive Control (RL-RNMPC) framework for controlling nonlinear systems in the presence of disturbances and uncertainties. An approximate Robust Nonlinear Model Predictive Control (RNMPC) of low computational complexity is used in which the state trajectory uncertainty is modelled via ellipsoids. Reinforcement Learning is then used in order to handle the ellipsoidal approximation and improve the closed-loop performance of the scheme by adjusting the MPC parameters generating the ellipsoids. The approach is tested on a simulated Wheeled Mobile Robot (WMR) tracking a desired trajectory while avoiding static obstacles.
In this paper, we discuss the deterministic policy gradient using the Actor-Critic methods based on the linear compatible advantage function approximator, where the input spaces are continuous. When the policy is restricted by hard constraints, the exploration may not be Centred or Isotropic (non-CI). As a result, the policy gradient estimation can be biased. We focus on constrained policies based on Model Predictive Control (MPC) schemes and to address the bias issue, we propose an approximate Robust MPC approach accounting for the exploration. The RMPC-based policy ensures that a Centered and Isotropic (CI) exploration is approximately feasible. A posterior projection is used to ensure its exact feasibility, we formally prove that this approach does not bias the gradient estimation.