A hybrid hydropower power plant is a conventional HydroPower Plant (HPP) augmented with a Battery Energy Storage System (BESS) to decrease the wear and tear of sensitive mechanical components and improve the reliability and regulation performance of the overall plant. A central task of controlling hybrid power plants is determining how the total power set-point should be split between the BESS and the hybridized unit (power set-point splitting) as a function of the operational objectives. This paper describes a Model Predictive Control (MPC) framework for hybrid medium- and high-head plants to determine the power set-point of the hydropower unit and the BESS. The splitting policy relies on an explicit formulation of the mechanical loads incurred by the HPP's penstock, which can be damaged due to fatigue when providing regulation services to the grid. By filtering out from the HPP's power set-point the components conducive to excess penstock fatigue and properly controlling the BESS, the proposed MPC is able to maintain the same level of regulation performance while significantly decreasing damages to the hydraulic conduits. A proof-of-concept by simulations is provided considering a 230 MW medium-head hydropower plant.
The paper presents a data-driven predictive control framework based on an implicit input-output mapping derived directly from the signal matrix of collected data. This signal matrix model is derived by maximum likelihood estimation with noise-corrupted data. By linearizing online, the implicit model can be used as a linear constraint to characterize possible trajectories of the system in receding horizon control. The signal matrix can also be updated online with new measurements. This algorithm can be applied to large datasets and slowly time-varying systems, possibly with high noise levels. An additional regularization term on the prediction error can be introduced to enhance the predictability and thus the control performance. Numerical results demonstrate that the proposed signal matrix model predictive control algorithm is effective in multiple applications and performs better than existing data-driven predictive control algorithm.
Model predictive control (MPC) is increasingly being considered for control of fast systems and embedded applications. However, the MPC has some significant challenges for such systems. Its high computational complexity results in high power consumption from the control algorithm, which could account for a significant share of the energy resources in battery-powered embedded systems. The MPC parameters must be tuned, which is largely a trial-and-error process that affects the control performance, the robustness and the computational complexity of the controller to a high degree. In this paper, we propose a novel framework in which any parameter of the control algorithm can be jointly tuned using reinforcement learning(RL), with the goal of simultaneously optimizing the control performance and the power usage of the control algorithm. We propose the novel idea of optimizing the meta-parameters of MPCwith RL, i.e. parameters affecting the structure of the MPCproblem as opposed to the solution to a given problem. Our control algorithm is based on an event-triggered MPC where we learn when the MPC should be re-computed, and a dual mode MPC and linear state feedback control law applied in between MPC computations. We formulate a novel mixture-distribution policy and show that with joint optimization we achieve improvements that do not present themselves when optimizing the same parameters in isolation. We demonstrate our framework on the inverted pendulum control task, reducing the total computation time of the control system by 36% while also improving the control performance by 18.4% over the best-performing MPC baseline.
We present a careful comparison of two model-free control algorithms, Evolution Strategies (ES) and Proximal Policy Optimization (PPO), with receding horizon model predictive control (MPC) for operating simulated, price responsive water heaters. Four MPC variants are considered: a one-shot controller with perfect forecasting yielding optimal control; a limited-horizon controller with perfect forecasting; a mean forecasting-based controller; and a two-stage stochastic programming controller using historical scenarios. In all cases, the MPC model for water temperature and electricity price are exact; only water demand is uncertain. For comparison, both ES and PPO learn neural network-based policies by directly interacting with the simulated environment under the same scenarios used by MPC. All methods are then evaluated on a separate one-week continuation of the demand time series. We demonstrate that optimal control for this problem is challenging, requiring more than 8-hour lookahead for MPC with perfect forecasting to attain the minimum cost. Despite this challenge, both ES and PPO learn good general purpose policies that outperform mean forecast and two-stage stochastic MPC controllers in terms of average cost and are more than two orders of magnitude faster at computing actions. We show that ES in particular can leverage parallelism to learn a policy in under 90 seconds using 1150 CPU cores.
Tube-based model predictive control (MPC) methods leverage tubes to bound deviations from a nominal trajectory due to uncertainties in order to ensure constraint satisfaction. This paper presents a novel tube-based MPC formulation based on system level disturbance reachable sets (SL-DRS), which leverage the affine system level parameterization (SLP). We show that imposing a finite impulse response (FIR) constraint on the affine SLP guarantees containment of all future deviations in a finite sequence of SL-DRS. This allows us to formulate a system level tube-MPC (SLTMPC) method using the SL-DRS as tubes, which enables concurrent optimization of the nominal trajectory and the tubes, while using a positively invariant terminal set. Finally, we show that the SL-DRS tubes can also be computed offline.
The displacement of conventional generation in favor of stochastic renewable requires increasing regulation duties from the remaining dispatchable resources. In high- and medium-head hydropower plants (HPPs), providing regulation services to the grid and frequently changing the plant's set-point causes water hammer, which engenders pressure and stress transients within the pressurized conduits, especially the penstock, damaging it in the long run. This paper proposes a model predictive control (MPC) that explicitly models the hydraulic transients within the penstock. It achieves to reduce the mechanical loads on the penstock wall, and, consequently, fatigue effectively. Thanks to a suitable linearization of the plant model, the optimization problem underlying the MPC scheme is convex and can be solved with off-the-shelf optimization libraries. The performance of the proposed controller is tested with numerical simulations on a 230MW medium-head HPP with Francis turbine providing primary frequency control. Simulation results show substantially reduced penstock fatigue, existing approaches outperformed, and problem resolution times compatible with real-time control requirements.
The rapid growth of renewable energy technology enables the concept of microgrid (MG) to be widely accepted in the power systems. Due to the advantages of the DC distribution system such as easy integration of energy storage and less system loss, DC MG attracts significant attention nowadays. The linear controller such as PI or PID is matured and extensively used by the power electronics industry, but their performance is not optimal as system parameters are changed. In this study, an artificial neural network (ANN) based voltage control strategy is proposed for the DC-DC boost converter. In this paper, the model predictive control (MPC) is used as an expert, which provides the data to train the proposed ANN. As ANN is tuned finely, then it is utilized directly to control the step-up DC converter. The main advantage of the ANN is that the neural network system identification decreases the inaccuracy of the system model even with inaccurate parameters and has less computational burden compared to MPC due to its parallel structure. To validate the performance of the proposed ANN, extensive MATLAB/Simulink simulations are carried out. The simulation results show that the ANN-based control strategy has better performance under different loading conditions comparison to the PI controller. The accuracy of the trained ANN model is about 97%, which makes it suitable to be used for DC microgrid applications.
Large-scale ride-hailing systems often combine real-time routing at the individual request level with a macroscopic Model Predictive Control (MPC) optimization for dynamic pricing and vehicle relocation. The MPC relies on a demand forecast and optimizes over a longer time horizon to compensate for the myopic nature of the routing optimization. However, the longer horizon increases computational complexity and forces the MPC to operate at coarser spatial-temporal granularity, degrading the quality of its decisions. This paper addresses these computational challenges by learning the MPC optimization. The resulting machine-learning model then serves as the optimization proxy and predicts its optimal solutions. This makes it possible to use the MPC at higher spatial-temporal fidelity, since the optimizations can be solved and learned offline. Experimental results show that the proposed approach improves quality of service on challenging instances from the New York City dataset.
This paper investigates the use of extended Kalman filtering to train recurrent neural networks with rather general convex loss functions and regularization terms on the network parameters, including $\ell_1$-regularization. We show that the learning method is competitive with respect to stochastic gradient descent in a nonlinear system identification benchmark and in training a linear system with binary outputs. We also explore the use of the algorithm in data-driven nonlinear model predictive control and its relation with disturbance models for offset-free closed-loop tracking.
Adverse circumstances such as extreme weather events can cause significant disruptions to normal operation of electric distribution systems (DS), which includes isolating parts of the DS due to damaged transmission equipment. In this paper, we consider the problem of load restoration in a microgrid (MG) that is islanded from the upstream DS. The MG contains sources of distributed generation such as microturbines and renewable energy sources, as well as energy storage systems (ESS). We formulate the load restoration task as a non-convex optimization problem. This problem embodies the physics of the MG by leveraging a branch flow model, while incorporating salient phenomenon in islanded MGs such as the need for internal frequency regulation, and complementarity requirements arising in ESS operations. Since the formulated optimization problem is non-convex, we introduce a convex relaxation which can be solved through model predictive control as a baseline method. However, in order to solve the problem considering its full non-convexity, we leverage a policy-learning method called constrained policy optimization, a tailored version of which is used as our proposed algorithm. The aforementioned approaches, along with an additional deep learning method are compared through extensive simulations.
The rapid adoption of electric vehicles (EVs) calls for the widespread installation of EV charging stations. To maximize the profitability of charging stations, intelligent controllers that provide both charging and electric grid services are in great need. However, it is challenging to determine the optimal charging schedule due to the uncertain arrival time and charging demands of EVs. In this paper, we propose a novel centralized allocation and decentralized execution (CADE) reinforcement learning (RL) framework to maximize the charging station's profit. In the centralized allocation process, EVs are allocated to either the waiting or charging spots. In the decentralized execution process, each charger makes its own charging/discharging decision while learning the action-value functions from a shared replay memory. This CADE framework significantly improves the scalability and sample efficiency of the RL algorithm. Numerical results show that the proposed CADE framework is both computationally efficient and scalable, and significantly outperforms the baseline model predictive control (MPC). We also provide an in-depth analysis of the learned action-value function to explain the inner working of the reinforcement learning agent.
Autonomous driving has attracted lots of attention in recent years. An accurate vehicle dynamics is important for autonomous driving techniques, e.g. trajectory prediction, motion planning, and control of trajectory tracking. Although previous works have made some results, the strong nonlinearity, precision, and interpretability of dynamics for autonomous vehicles are open problems worth being studied. In this paper, the approach based on the Koopman operator named deep direct Koopman (DDK) is proposed to identify the model of the autonomous vehicle and the identified model is a linear time-invariant (LTI) version, which is convenient for motion planning and controller design. In the approach, the Koopman eigenvalues and system matrix are considered as trainable tensors with the original states of the autonomous vehicle being concatenated to a part of the Koopman eigenfunctions so that a physically interpretable subsystem can be extracted from the identified latent dynamics. Subsequently, the process of the identification model is trained under the proposed method based on the dataset which consists of about 60km of data collected with a real electric SUV while the effectiveness of the identified model is validated. Meanwhile, a high-fidelity vehicle dynamics is identified in CarSim with DDK, and then, a linear model predictive control (MPC) called DDK-MPC integrating DDK is designed to validate the performance for the control of trajectory tracking. Simulation results illustrate that the model of the nonlinear vehicle dynamics can be identified effectively via the proposed method and that excellent tracking performance can be obtained with the identified model under DDK-MPC.
Various efforts have been devoted to developing stabilizing distributed Model Predictive Control (MPC) schemes for tracking piecewise constant references. In these schemes, terminal sets are usually computed offline and used in the MPC online phase to guarantee recursive feasibility and asymptotic stability. Maximal invariant terminal sets do not necessarily respect the distributed structure of the network, hindering the distributed implementation of the controller. On the other hand, ellipsoidal terminal sets respect the distributed structure, but may lead to conservative schemes. In this paper, a novel distributed MPC scheme is proposed for reference tracking of networked dynamical systems where the terminal ingredients are reconfigured online depending on the closed-loop states to alleviate the aforementioned issues. The resulting non-convex infinite-dimensional problem is approximated using a quadratic program. The proposed scheme is tested in simulation where the proposed MPC problem is solved using distributed optimization.
Energy requirements for heating and cooling of buildings constitute a major fraction of end use energy consumed. Therefore, it is important to provide the occupant comfort requirements in buildings in an energy efficient manner. However, buildings are large scale complex systems, susceptible to sensor, actuator or communication network failures in their thermal control infrastructure, that can affect their performance in terms of occupant comfort and energy efficiency. The degree of decentralization in the control architecture determines a fundamental tradeoff between performance and robustness. This thesis studies the problem of thermal control of buildings from the perspective of partitioning them into clusters for decentralized control, to balance underlying performance and robustness requirements. Measures of deviation in performance and robustness between centralized and decentralized architectures in the Model Predictive Control framework are derived. Appropriate clustering algorithms are then proposed to determine decentralized control architectures which provide a satisfactory trade-off between the underlying performance and robustness objectives. Two different partitioning methodologies the CLF-MCS method and the OLF-FPM method are developed and compared. The problem of decentralized control design based on the architectures obtained using these methodologies is also considered. It entails the use of decentralized extended state observers to address the issue of unavailability of unknown states and disturbances in the system. The potential use of the proposed control architecture selection and decentralized control design methodologies is demonstrated in simulation on a real world multi-zone building.
We provide a functional view of distributional robustness motivated by robust statistics and functional analysis. This results in two practical computational approaches for approximate distributionally robust nonlinear optimization based on gradient norms and reproducing kernel Hilbert spaces. Our method can be applied to the settings of statistical learning with small sample size and test distribution shift. As a case study, we robustify scenario-based stochastic model predictive control with general nonlinear constraints. In particular, we demonstrate constraint satisfaction with only a small number of scenarios under distribution shift.
Active building energy management holds potential to reduce global energy-related emissions and support flexible operations of future low-carbon systems. This requires to integrate diverse objectives and engage multiple stakeholders. However, there remains a gap in comprehensive field insights into emission reduction, flexibility provision, and user impacts. This study examined how a real occupied building, with all its energy assets, could function as an emission-aware flexible prosumer. An existing building energy management system was enhanced by integrating a model predictive control strategy. The enhanced setup minimized the equivalent carbon emission due to electricity imports and provided flexibility to the energy system. The experimental results indicated an emission reduction of 12.5% compared to a rule-based controller that maximized PV self-consumption. In addition, a minimal flexibility provision experiment was demonstrated with a locally emulated distribution system operator. The results suggested that flexibility was provided without the risk of rebound effects. This is due to the flexibility envelope that was self-reported in advance. The study concluded by highlighting technical challenges in realizing emission reduction and flexibility in practice.
The study of resilient control of linear time-invariant (LTI) systems against denial-of-service (DoS) attacks is gaining popularity in emerging cyber-physical applications. In previous works, explicit system models are required to design a predictor-based resilient controller. These models can be either given a priori or obtained through a prior system identification step. Recent research efforts have focused on data-driven control based on pre-collected input-output trajectories (i.e., without explicit system models). In this paper, we take an initial step toward data-driven stabilization of stochastic LTI systems under DoS attacks, and develop a resilient model predictive control (MPC) scheme driven purely by data-dependent conditions. The proposed data-driven control method achieves the same level of resilience as the model-based control method. For example, local input-to-state stability (ISS) is achieved under mild assumptions on the noise and the DoS attacks. To recover global ISS, two modifications are further suggested at the price of reduced resilience against DoS attacks or increased computational complexity. Finally, a numerical example is given to validate the effectiveness of the proposed control method.
Recent works in Reinforcement Learning (RL) combine model-free (Mf)-RL algorithms with model-based (Mb)-RL approaches to get the best from both: asymptotic performance of Mf-RL and high sample-efficiency of Mb-RL. Inspired by these works, we propose a hierarchical framework that integrates online learning for the Mb-trajectory optimization with off-policy methods for the Mf-RL. In particular, two loops are proposed, where the Dynamic Mirror Descent based Model Predictive Control (DMD-MPC) is used as the inner loop to obtain an optimal sequence of actions. These actions are in turn used to significantly accelerate the outer loop Mf-RL. We show that our formulation is generic for a broad class of MPC based policies and objectives, and includes some of the well-known Mb-Mf approaches. Based on the framework we define two algorithms to increase sample efficiency of Off Policy RL and to guide end to end RL algorithms for online adaption respectively. Thus we finally introduce two novel algorithms: Dynamic-Mirror Descent Model Predictive RL(DeMoRL), which uses the method of elite fractions for the inner loop and Soft Actor-Critic (SAC) as the off-policy RL for the outer loop and Dynamic-Mirror Descent Model Predictive Layer(DeMo Layer), a special case of the hierarchical framework which guides linear policies trained using Augmented Random Search(ARS). Our experiments show faster convergence of the proposed DeMo RL, and better or equal performance compared to other Mf-Mb approaches on benchmark MuJoCo control tasks. The DeMo Layer was tested on classical Cartpole and custom-built Quadruped trained using Linear Policy.
We propose a computationally efficient approach to safe reinforcement learning (RL) for frequency regulation in power systems with high levels of variable renewable energy resources. The approach draws on set-theoretic control techniques to craft a neural network-based control policy that is guaranteed to satisfy safety-critical state constraints, without needing to solve a model predictive control or projection problem in real time. By exploiting the properties of robust controlled-invariant polytopes, we construct a novel, closed-form "safety-filter" that enables end-to-end safe learning using any policy gradient-based RL algorithm. We then apply the safety filter in conjunction with the deep deterministic policy gradient (DDPG) algorithm to regulate frequency in a modified 9-bus power system, and show that the learned policy is more cost-effective than robust linear feedback control techniques while maintaining the same safety guarantee. We also show that the proposed paradigm outperforms DDPG augmented with constraint violation penalties.
Among the auxiliary loads in light-duty vehicles, the air conditioning system is the single largest energy consumer. For electrified vehicles, the impact of heating and cooling loads becomes even more significant, as they compete with the powertrain for battery energy use and can significantly reduce the range or performance. While considerable work has been made in the field of optimal energy management for electrified vehicles and optimization of vehicle velocity for eco-driving, few contributions have addressed the application of energy-optimal control for heating and cooling loads. This paper proposes an energy management strategy for the thermal management system of an electrified powertrain, based on Model Predictive Control. Starting from a nonlinear model of the vapor compression refrigeration system that captures the dynamics of the refrigerant in the heat exchangers and the power consumption of the system, a constrained multi-objective optimal control problem is formulated to reduce energy consumption while tracking a desired thermal set point. An efficient implementation of MPC is proposed for real-time applications by introducing a terminal cost obtained from the approximation of the global optimal solution.