We propose an adaptive optimisation approach for tuning stochastic model predictive control (MPC) hyper-parameters while jointly estimating probability distributions of the transition model parameters based on performance rewards. In particular, we develop a Bayesian optimisation (BO) algorithm with a heteroscedastic noise model to deal with varying noise across the MPC hyper-parameter and dynamics model parameter spaces. Typical homoscedastic noise models are unrealistic for tuning MPC since stochastic controllers are inherently noisy, and the level of noise is affected by their hyper-parameter settings. We evaluate the proposed optimisation algorithm in simulated control and robotics tasks where we jointly infer control and dynamics parameters. Experimental results demonstrate that our approach leads to higher cumulative rewards and more stable controllers.
This article presents a distributed model-predictive control (MPC) design for real-time voltage control in power systems, including an online method to estimate the bus admittance matrix $\mathbf{Y}$ to let it be time-varying and unknown a priori. The prevalent control designs are either (a) centralized, providing optimal solutions but less scalable and susceptible to single-point failures/attacks, or (b) decentralized or localized, having increased scalability and attack resilience but are suboptimal. The proposed distributed solution offers the attractive features of both methodologies, where neighboring nodes share state information to attain a globally optimal solution. In addition, the presented framework provides a data-driven estimation of Y to circumvent the challenging issue of acquiring accurate knowledge of the line impedance (required to form Y). We first introduce the centralized version of the predictive voltage control problem and then transfer it to a distributed version. The distributed version is solved via the alternating direction method of multipliers (ADMM), using only local measurements and communication leveraging the graph structure of the power system. The proposed framework is resilient to prediction uncertainty, modeling error, and communication link failure, and the in-built redundancy within the proposed framework supports anomaly detection in cyberattacks. We validate the proposed methodology for IEEE-30 bus, IEEE-57 bus transmission systems, and IEEE-123 bus distribution systems.
Laser photocoagulation is one of the most frequently used treatment approaches for retinal diseases such as diabetic retinopathy and macular edema. The use of model-based control, such as Model Predictive Control (MPC), enhances a safe and effective treatment by guaranteeing temperature bounds. In general, real-time requirements for model-based control designs are not met since the temperature distribution in the eye fundus is governed by a heat equation with a nonlinear parameter dependency. This issue is circumvented by representing the model by a lower-dimensional system which well-approximates the original model, including the parametric dependency. We combine a global-basis approach with the discrete empirical interpolation method, tailor its hyperparameters to laser photocoagulation, and show its superiority in comparison to a recently proposed method based on Taylor-series approximation. Its effectiveness is measured in computation time for MPC. We further present a case study to estimate the range of absorption parameters in porcine eyes, and by means of a theoretical and numerical sensitivity analysis we show that the sensitivity of the temperature increase is higher with respect to the absorption coefficient of the retinal pigment epithelium (RPE) than of the choroid's.
Laser photocoagulation is a technique applied in the treatment of retinal diseases. While this is often done manually or using simple control schemes, we pursue an optimization-based approach, namely Model Predictive Control (MPC), to enforce bounds on the peak temperature and, thus, safety during the medical treatment procedure - despite the spot-dependent absorption of the tissue. To this end, a repetition rate of 1 kHz is desirable rendering the real-time requirements a major challenge. We present a tailored MPC scheme using parametric model reduction, an extended Kalman filter for the parameter and state estimation, and suitably constructed stage costs and verify its applicability both in simulation and experiments with porcine eyes. Moreover, we give some insight on the implementation specifically tailored for fast numerical computations.
In this paper, we present a data-driven distributed model predictive control (MPC) scheme to stabilise the origin of dynamically coupled discrete-time linear systems subject to decoupled input constraints. The local optimisation problems solved by the subsystems rely on a distributed adaptation of the Fundamental Lemma by Willems et al., allowing to parametrise system trajectories using only measured input-output data without explicit model knowledge. For the local predictions, the subsystems rely on communicated assumed trajectories of neighbours. Each subsystem guarantees a small deviation from these trajectories via a consistency constraint. We provide a theoretical analysis of the resulting non-iterative distributed MPC scheme, including proofs of recursive feasibility and (practical) stability. Finally, the approach is successfully applied to a numerical example.
This paper proposes a stabilising model predictive control (MPC) scheme with preview information of disturbance for nonlinear systems. The proposed MPC algorithm is able to not only reject disturbance by making use of disturbance preview information as necessary, but also take advantage of the disturbance if it is good for a control task. This is realised by taking into account both the task (e.g. reference trajectory) and disturbance preview in the prediction horizon when performing online optimisation. Conditions are established to ensure recursive feasibility and stability under the disturbance. First the disturbance within the horizon is augmented with the state to form a new composite system and then the stage cost function is modified accordingly. With the help of input-to-state stability theory, a terminal cost and a terminal constraint are constructed and added to the MPC algorithm with preview to guarantee its recursive feasibility and stability under a pre-bounded disturbance. Numerical simulation results demonstrate the effectiveness of the proposed MPC algorithm.
In this paper, two novel practical methods of Reinforcement Learning from Demonstration (RLfD) are developed and applied to automatic berthing control systems for Unmanned Surface Vessel. A new expert data generation method, called Model Predictive Based Expert (MPBE) which combines Model Predictive Control and Deep Deterministic Policy Gradient, is developed to provide high quality supervision data for RLfD algorithms. A straightforward RLfD method, model predictive Deep Deterministic Policy Gradient (MP-DDPG), is firstly introduced by replacing the RL agent with MPBE to directly interact with the environment. Then distribution mismatch problem is analyzed for MP-DDPG, and two techniques that alleviate distribution mismatch are proposed. Furthermore, another novel RLfD algorithm based on the MP-DDPG, called Self-Guided Actor-Critic (SGAC) is present, which can effectively leverage MPBE by continuously querying it to generate high quality expert data online. The distribution mismatch problem leading to unstable learning process is addressed by SGAC in a DAgger manner. In addition, theoretical analysis is given to prove that SGAC algorithm can converge with guaranteed monotonic improvement. Simulation results verify the effectiveness of MP-DDPG and SGAC to accomplish the ship berthing control task, and show advantages of SGAC comparing with other typical reinforcement learning algorithms and MP-DDPG.
A hierarchical Model Predictive Control (MPC) formulation is presented for coupled discrete-time linear systems with state and input constraints. Compared to a centralized approach, a two-level hierarchical controller, with one controller in the upper-level and one controller per subsystem in the lower-level, can significantly reduce the computational cost associated with MPC. Hierarchical coordination is achieved using adjustable tubes, which are optimized by the upper-level controller and bound permissible lower-level controller deviations from the system trajectories determined by the upper-level controller. The size of these adjustable tubes determines the degree of uncertainty between subsystems and directly affects the required constraint tightening under a tube-based robust MPC framework. Sets are represented as zonotopes to enable the ability to optimize the size of these adjustable tubes and perform the necessary constraint tightening online as part of the MPC optimization problems. State and input constraint satisfaction is proven for the two-level hierarchical controller with an arbitrary number of controllers at the lower-level and a numerical example demonstrates the key features and performance of the approach.
The paper deals with the design of an improved model predictive control scheme for achieving station-keeping in a quasi Halo orbit around the $L_2$ point in the Earth-Moon system. The improvement is obtained thanks to a multi-rate sampled-data trajectory planner that allows for simplifying the optimization problem of the model predictive controller while guaranteeing feasibility and convergence to the desired orbit. The multi-rate planner is designed based on a simplified model of the dynamics under a preliminary nonlinear regulation feedback. The proposed control scheme is shown to outperform recent station-keeping nonlinear model predictive control designs both in terms of tracking error and energy expenditures in different situations. Finally, a brief study of aspects pertaining to computational time are carried out so highlighting the possibility for real time implementation on modern hardware.
We model a platooning system including trucks and a third-party service provider that performs platoon coordination, distributes the platooning profit within platoons, and charges the trucks in exchange for its services. This paper studies one class of pricing rules, where the third-party service provider keeps part of the platooning profit each time a platoon is formed. Furthermore, we propose a platoon coordination solution based on distributed model predictive control in which the pricing rule is integrated. To evaluate the effect of the pricing on the platooning system, we perform a simulation over the Swedish road network. The simulation shows that the platooning rate and profit highly depend on the pricing. This suggests that pricing needs to be set carefully to obtain a satisfactory platooning system in the future.
A state-space representation of water quality dynamics describing disinfectant (e.g., chlorine) transport dynamics in drinking water distribution networks has been recently proposed. Such representation is a byproduct of space- and time-discretization of the PDE modeling transport dynamics. This results in a large state-space dimension even for small networks with tens of nodes. Although such a state-space model provides a model-driven approach to predict water quality dynamics, incorporating it into model-based control algorithms or state estimators for large networks is challenging and at times intractable. To that end, this paper investigates model order reduction (MOR) methods for water quality dynamics with the objective of performing post-reduction feedback control. The presented investigation focuses on reducing state-dimension by orders of magnitude, the stability of the MOR methods, and the application of these methods to model predictive control.
This paper considers the problem of hub-based platoon coordination for a large-scale transport system, where trucks have individual utility functions to optimize. An event-triggered distributed model predictive control method is proposed to solve the optimal scheduling of waiting times at hubs for individual trucks. In this distributed framework, trucks are allowed to decide their waiting times independently and only limited information is shared between trucks. Both the predicted reward gained from platooning and the predicted cost for waiting at hubs are included in each truck's utility function. The performance of the coordination method is demonstrated in a simulation with one hundred trucks over the Swedish road network.
The ability to accurately predict human behavior is central to the safety and efficiency of robot autonomy in interactive settings. Unfortunately, robots often lack access to key information on which these predictions may hinge, such as people's goals, attention, and willingness to cooperate. Dual control theory addresses this challenge by treating unknown parameters of a predictive model as stochastic hidden states and inferring their values at runtime using information gathered during system operation. While able to optimally and automatically trade off exploration and exploitation, dual control is computationally intractable for general interactive motion planning, mainly due to the fundamental coupling between robot trajectory optimization and human intent inference. In this paper, we present a novel algorithmic approach to enable active uncertainty reduction for interactive motion planning based on the implicit dual control paradigm. Our approach relies on sampling-based approximation of stochastic dynamic programming, leading to a model predictive control problem that can be readily solved by real-time gradient-based optimization methods. The resulting policy is shown to preserve the dual control effect for a broad class of predictive human models with both continuous and categorical uncertainty. The efficacy of our approach is demonstrated with simulated driving examples.
This paper presents a novel, safe control architecture (SCA) for controlling an important class of systems: safety-critical systems. Ensuring the safety of control decisions has always been a challenge in automatic control. The proposed SCA aims to address this challenge by using a Model Predictive Controller (MPC) that acts as a supervisor for the operating controller, in the sense that the MPC constantly checks the safety of the control inputs generated by the operating controller and intervenes if the control input is predicted to lead to a hazardous situation in the foreseeable future invariably. Then an appropriate backup scheme can be activated, e.g., a degraded control mechanism, the transfer of the system to a safe state, or a warning signal issued to a human supervisor. For a proof of concept, the proposed SCA is applied to an autonomous driving scenario, where it is illustrated and compared in different obstacle avoidance scenarios. A major challenge of the SCA lies in the mismatch between the MPC prediction model and the real system, for which possible remedies are explored.
Reinforcement learning (RL) is a promising, upcoming topic in automatic control applications. Where classical control approaches require a priori system knowledge, data-driven control approaches like RL allow a model-free controller design procedure, rendering them emergent techniques for systems with changing plant structures and varying parameters. While it was already shown in various applications that the transient control behavior for complex systems can be sufficiently handled by RL, the challenge of non-vanishing steady-state control errors remains, which arises from the usage of control policy approximations and finite training times. To overcome this issue, an integral action state augmentation (IASA) for actor-critic-based RL controllers is introduced that mimics an integrating feedback, which is inspired by the delta-input formulation within model predictive control. This augmentation does not require any expert knowledge, leaving the approach model free. As a result, the RL controller learns how to suppress steady-state control deviations much more effectively. Two exemplary applications from the domain of electrical energy engineering validate the benefit of the developed method both for reference tracking and disturbance rejection. In comparison to a standard deep deterministic policy gradient (DDPG) setup, the suggested IASA extension allows to reduce the steady-state error by up to 52 $\%$ within the considered validation scenarios.
Offline Reinforcement Learning (ORL) enablesus to separately study the two interlinked processes of reinforcement learning: collecting informative experience and inferring optimal behaviour. The second step has been widely studied in the offline setting, but just as critical to data-efficient RL is the collection of informative data. The task-agnostic setting for data collection, where the task is not known a priori, is of particular interest due to the possibility of collecting a single dataset and using it to solve several downstream tasks as they arise. We investigate this setting via curiosity-based intrinsic motivation, a family of exploration methods which encourage the agent to explore those states or transitions it has not yet learned to model. With Explore2Offline, we propose to evaluate the quality of collected data by transferring the collected data and inferring policies with reward relabelling and standard offline RL algorithms. We evaluate a wide variety of data collection strategies, including a new exploration agent, Intrinsic Model Predictive Control (IMPC), using this scheme and demonstrate their performance on various tasks. We use this decoupled framework to strengthen intuitions about exploration and the data prerequisites for effective offline RL.
Human-leading truck platooning systems have been proposed to leverage the benefits of both human supervision and vehicle autonomy. Equipped with human guidance and autonomous technology, human-leading truck platooning systems are more versatile to handle uncertain traffic conditions than fully automated platooning systems. This paper presents a novel distributed stochastic model predictive control (DSMPC) design for a human-leading heavy-duty truck platoon. The proposed DSMPC design integrates the stochastic driver behavior model of the human-driven leader truck with a distributed formation control design for the following automated trucks in the platoon. The driver behavior of the human-driven leader truck is learned by a stochastic inverse reinforcement learning (SIRL) approach. The proposed stochastic driver behavior model aims to learn a distribution of cost function, which represents the richness and uniqueness of human driver behaviors, with a given set of driver-specific demonstrations. The distributed formation control includes a serial DSMPC with guaranteed recursive feasibility, closed-loop chance constraint satisfaction, and string stability. Simulation studies are conducted to investigate the efficacy of the proposed design under several realistic traffic scenarios. Compared to the baseline platoon control strategy (deterministic distributed model predictive control), the proposed DSMPC achieves superior controller performance in constraint violations and spacing errors.
Based on a series of two experimental campaigns testing advanced controllers on a scaled wind turbine operating in a wind tunnel, this contribution describes the overall experimental method, challenges faced, lessons learned, and opportunities for future work. The two campaigns, run in Fall 2018 and Fall 2019, tested unconstrained and constrained optimal blade pitch controllers, respectively, using preview disturbance measurements of the oncoming wind. Specifically, the first study considered an extension to the linear-quadratic regulator to include feedforward action, while the second deployed model predictive control to incorporate actuator constraints into the optimal control problem. The results of the campaigns have already been published in technical conference and journal papers on control systems; however, detail on how the controllers were implemented was not included in those works. We aim to fill that gap with this contribution, which is targeted at the wind energy community. We describe several aspects of the experimental setup, in particular providing details of the software and hardware used for the controller; share insight on several aspects of the procedure that were difficult and how we overcame those challenges; and summarize the key differences between simulation-based studies and physical testing. By doing so, we hope to share what we learned during our experimental campaigns and provide a point of reference for others looking to carry out experiments on scaled wind turbines operating in wind tunnel facilities.
Lane change for autonomous vehicles (AVs) is an important but challenging task in complex dynamic traffic environments. Due to difficulties in guarantee safety as well as a high efficiency, AVs are inclined to choose relatively conservative strategies for lane change. To avoid the conservatism, this paper presents a cooperation-aware lane change method utilizing interactions between vehicles. We first propose an interactive trajectory prediction method to explore possible cooperations between an AV and the others. Further, an evaluation is designed to make a decision on lane change, in which safety, efficiency and comfort are taken into consideration. Thereafter, we propose a motion planning algorithm based on model predictive control (MPC), which incorporates AV's decision and surrounding vehicles' interactive behaviors into constraints so as to avoid collisions during lane change. Quantitative testing results show that compared with the methods without an interactive prediction, our method enhances driving efficiencies of the AV and other vehicles by 14.8$\%$ and 2.6$\%$ respectively, which indicates that a proper utilization of vehicle interactions can effectively reduce the conservatism of the AV and promote the cooperation between the AV and others.
Satellite constellation missions, consisting of a large number of spacecraft, are increasingly being launched or planned. Such missions require novel control approaches, in particular for what concerns orbital phasing maneuvers. In this context, we consider the problem of reconfiguration of a satellite constellation in a circular formation. In our scenario, a formation of equally spaced spacecraft need to undergo an autonomous reconfiguration due to the deorbiting of a satellite in the formation. The remaining spacecraft have to reconfigure to form again an equidistant formation. To achieve this goal, we consider two decentralized strategies that rely on different sets of information about the neighboring spacecraft in the formation. In the fully decentralized case, each controller knows only the current states of each spacecraft, i.e. position and velocity, while in the second decentralized strategy with with information sharing, the entire planned nominal trajectory of each spacecraft is available to its neighbors. Our numerical simulation results show that, by increasing the amount of information available to each spacecraft, faster reconfiguration maneuvers with smaller fuel consumption can be achieved.