Infinite-horizon optimal control of constrained piecewise affine (PWA) systems has been approximately addressed by hybrid model predictive control (MPC), which, however, has computational limitations, both in offline design and online implementation. In this paper, we consider an alternative approach based on approximate dynamic programming (ADP), an important class of methods in reinforcement learning. We accommodate non-convex union-of-polyhedra state constraints and linear input constraints into ADP by designing PWA penalty functions. PWA function approximation is used, which allows for a mixed-integer encoding to implement ADP. The main advantage of the proposed ADP method is its online computational efficiency. Particularly, we propose two control policies, which lead to solving a smaller-scale mixed-integer linear program than conventional hybrid MPC, or a single convex quadratic program, depending on whether the policy is implicitly determined online or explicitly computed offline. We characterize the stability and safety properties of the closed-loop systems, as well as the sub-optimality of the proposed policies, by quantifying the approximation errors of value functions and policies. We also develop an offline mixed-integer linear programming-based method to certify the reliability of the proposed method. Simulation results on an inverted pendulum with elastic walls and on an adaptive cruise control problem validate the control performance in terms of constraint satisfaction and CPU time.
Providing an execution time certificate is a pressing requirement when deploying Model Predictive Control (MPC) in real-time embedded systems such as microcontrollers. Real-time MPC requires that its worst-case (maximum) execution time must be theoretically guaranteed to be smaller than the sampling time in closed-loop. This technical note considers input-constrained MPC problems and exploits the structure of the resulting box-constrained QPs. Then, we propose a \textit{cost-free} and \textit{data-independent} initialization strategy, which enables us, for the first time, to remove the initialization assumption of feasible full-Newton interior-point algorithms. We prove that the number of iterations of our proposed algorithm is \textit{only dimension-dependent} (\textit{data-independent}), \textit{simple-calculated}, and \textit{exact} (not \textit{worst-case}) with the value $\left\lceil\frac{\log(\frac{2n}{\epsilon})}{-2\log(\frac{\sqrt{2n}}{\sqrt{2n}+\sqrt{2}-1})}\right\rceil \!+ 1$, where $n$ denotes the problem dimension and $\epsilon$ denotes the constant stopping tolerance. These features enable our algorithm to trivially certify the execution time of nonlinear MPC (via online linearized schemes) or adaptive MPC problems. The execution-time-certified capability of our algorithm is theoretically and numerically validated through an open-loop unstable AFTI-16 example.
This work investigates traffic control via controlled connected and automated vehicles (CAVs) using novel controllers derived from the linear-quadratic regulator (LQR) theory. CAV-platoons are modeled as moving bottlenecks impacting the surrounding traffic with their speeds as control inputs. An iterative controller algorithm based on the LQR theory is proposed along with a variant that allows for penalizing abrupt changes in platoons speeds. The controllers use the Lighthill-Whitham-Richards (LWR) model implemented using an extended cell transmission model (CTM) which considers the capacity drop phenomenon for a realistic representation of traffic in congestion. The impact of various parameters of the proposed controller on the control performance is analyzed. The effectiveness of the proposed traffic control algorithms is tested using a traffic control example and compared with existing proportional-integral (PI)- and model predictive control (MPC)- based controllers from the literature. A case study using the TransModeler traffic microsimulation software is conducted to test the usability of the proposed controller as well as existing controllers in a realistic setting and derive qualitative insights. It is observed that the proposed controller works well in both settings to mitigate the impact of the jam caused by a fixed bottleneck. The computation time required by the controller is also small making it suitable for real-time control.
This paper considers a single-trajectory system identification problem for linear systems under general nonlinear and/or time-varying policies with i.i.d. random excitation noises. The problem is motivated by safe learning-based control for constrained linear systems, where the safe policies during the learning process are usually nonlinear and time-varying for satisfying the state and input constraints. In this paper, we provide a non-asymptotic error bound for least square estimation when the data trajectory is generated by any nonlinear and/or time-varying policies as long as the generated state and action trajectories are bounded. This significantly generalizes the existing non-asymptotic guarantees for linear system identification, which usually consider i.i.d. random inputs or linear policies. Interestingly, our error bound is consistent with that for linear policies with respect to the dependence on the trajectory length, system dimensions, and excitation levels. Lastly, we demonstrate the applications of our results by safe learning with robust model predictive control and provide numerical analysis.
The high concentration level of the environmental factors, such as a high ammonia concentration and pH level, affect the water quality, affecting fish's survival and mass death. Therefore, there is a critical need to develop control strategies to determine optimal, efficient, and reliable feeding and water quality monitoring processes. In this paper, we revisit the representative fish growth model describing the total biomass change by incorporating the fish population density and mortality. Since the measurement data of the total biomass and population from the aquaculture systems are limited and difficult to obtain, we validate the new dynamic population model with the individual fish growth data for tracking control purposes. We specifically focus on relative feeding as a manipulated variable to design traditional and optimal control to track the desired weight reference within the sub-optimal temperature and dissolved oxygen profiles under different levels of unionized ammonia exposure. Then, we propose a Q-learning approach that learns an optimal feeding control policy from the simulated data of the fish growth weight trajectories while managing the ammonia effects. The proposed Q-learning feeding control prevents fish mortality and achieves good tracking errors of the fish weight under the different levels of unionized ammonia. However, it maintains a relative food consumption that potentially underfeeds the fish. Finally, we propose an optimal algorithm that optimizes the feeding and water quality of the dynamic fish population growth process. We also show that the model predictive control decreases fish mortality and reduces food consumption in all different cases of unionized ammonia exposure.
One of the major challenges in the development of energy management systems (EMSs) for complex buildings is accurate modeling. To address this, we propose an EMS, which combines a Model Predictive Control (MPC) approach with data-driven model error compensation. The hierarchical MPC approach consists of two layers: An aggregator controls the overall energy flows of the building in an aggregated perspective, while a distributor distributes heating and cooling powers to individual temperature zones. The controllers of both layers employ regression-based error estimation to predict and incorporate the model error. The proposed approach is evaluated in a software-in-the-loop simulation using a physics-based digital twin model. Simulation results show the efficacy and robustness of the proposed approach
Learned visual dynamics models have proven effective for robotic manipulation tasks. Yet, it remains unclear how best to represent scenes involving multi-object interactions. Current methods decompose a scene into discrete objects, but they struggle with precise modeling and manipulation amid challenging lighting conditions as they only encode appearance tied with specific illuminations. In this work, we propose using object-centric neural scattering functions (OSFs) as object representations in a model-predictive control framework. OSFs model per-object light transport, enabling compositional scene re-rendering under object rearrangement and varying lighting conditions. By combining this approach with inverse parameter estimation and graph-based neural dynamics models, we demonstrate improved model-predictive control performance and generalization in compositional multi-object environments, even in previously unseen scenarios and harsh lighting conditions.
The agricultural sector currently faces significant challenges in water resource conservation and crop yield optimization, primarily due to concerns over freshwater scarcity. Traditional irrigation scheduling methods often prove inadequate in meeting the needs of large-scale irrigation systems. To address this issue, this paper proposes a predictive irrigation scheduler that leverages the three paradigms of machine learning to optimize irrigation schedules. The proposed scheduler employs the k-means clustering approach to divide the field into distinct irrigation management zones based on soil hydraulic parameters and topology information. Furthermore, a long short-term memory network is employed to develop dynamic models for each management zone, enabling accurate predictions of soil moisture dynamics. Formulated as a mixed-integer model predictive control problem, the scheduler aims to maximize water uptake while minimizing overall water consumption and irrigation costs. To tackle the mixed-integer optimization challenge, the proximal policy optimization algorithm is utilized to train a reinforcement learning agent responsible for making daily irrigation decisions. To evaluate the performance of the proposed scheduler, a 26.4-hectare field in Lethbridge, Canada, was chosen as a case study for the 2015 and 2022 growing seasons. The results demonstrate the superiority of the proposed scheduler compared to a traditional irrigation scheduling method in terms of water use efficiency and crop yield improvement for both growing seasons. Notably, the proposed scheduler achieved water savings ranging from 6.4% to 22.8%, along with yield increases ranging from 2.3% to 4.3%.
We present a framework and algorithms to learn controlled dynamics models using neural stochastic differential equations (SDEs) -- SDEs whose drift and diffusion terms are both parametrized by neural networks. We construct the drift term to leverage a priori physics knowledge as inductive bias, and we design the diffusion term to represent a distance-aware estimate of the uncertainty in the learned model's predictions -- it matches the system's underlying stochasticity when evaluated on states near those from the training dataset, and it predicts highly stochastic dynamics when evaluated on states beyond the training regime. The proposed neural SDEs can be evaluated quickly enough for use in model predictive control algorithms, or they can be used as simulators for model-based reinforcement learning. Furthermore, they make accurate predictions over long time horizons, even when trained on small datasets that cover limited regions of the state space. We demonstrate these capabilities through experiments on simulated robotic systems, as well as by using them to model and control a hexacopter's flight dynamics: A neural SDE trained using only three minutes of manually collected flight data results in a model-based control policy that accurately tracks aggressive trajectories that push the hexacopter's velocity and Euler angles to nearly double the maximum values observed in the training dataset.
Near the limits of adhesion, the forces generated by a tire are nonlinear and intricately coupled. Efficient and accurate modelling in this region could improve safety, especially in emergency situations where high forces are required. To this end, we propose a novel family of tire force models based on neural ordinary differential equations and a neural-ExpTanh parameterization. These models are designed to satisfy physically insightful assumptions while also having sufficient fidelity to capture higher-order effects directly from vehicle state measurements. They are used as drop-in replacements for an analytical brush tire model in an existing nonlinear model predictive control framework. Experiments with a customized Toyota Supra show that scarce amounts of driving data -- less than three minutes -- is sufficient to achieve high-performance autonomous drifting on various trajectories with speeds up to 45mph. Comparisons with the benchmark model show a $4 \times$ improvement in tracking performance, smoother control inputs, and faster and more consistent computation time.
This article proposes and evaluates a new safety concept called backup plan safety for path planning of autonomous vehicles under mission uncertainty using model predictive control (MPC). Backup plan safety is defined as the ability to complete an alternative mission when the primary mission is aborted. To include this new safety concept in control problems, we formulate a feasibility maximization problem aiming to maximize the feasibility of the primary and alternative missions. The feasibility maximization problem is based on multi-objective MPC, where each objective (cost function) is associated with a different mission and balanced by a weight vector. Furthermore, the feasibility maximization problem incorporates additional control input horizons toward the alternative missions on top of the control input horizon toward the primary mission, denoted as multi-horizon inputs, to evaluate the cost for each mission. We develop the backup plan constrained MPC algorithm, which designs the weight vector that ensures asymptotic stability of the closed-loop system, and generates the optimal control input by solving the feasibility maximization problem with computational efficiency. The performance of the proposed algorithm is validated through simulations of a UAV path planning problem.
This paper studies the design of a Model Predictive Controller (MPC) for integrated lateral stability, traction/braking control, and rollover prevention of electric vehicles intended for very high speed (VHS) racing applications. We first identify the advantages of a state-of-the-art dynamic model in that it includes rollover prevention into the MPC (a total of 8 states) and also linearizes the tire model prior to solving the MPC problem to save computation time. Then the design of a novel model predictive controller for lateral stability control is proposed aimed for achieving stable control at top speed significantly greater than typical highway speed limits. We have tested the new solution in simulation environments associated with the Indy Autonomous Challenge, where its real-world racing conditions include significant road banking angles, lateral position tracking, and a different suspension model of its Dallara Indy Lights chassis. The results are very promising with a low solver time in Python, as low as 50 Hz, and a lateral error of 30 cm at speeds of 45 m/s. Our open source code is available at: https: //github.com/jadyahya/Roll-Yaw-and-Lateral-Velocity-MPC/.
AI and machine learning based approaches are becoming ubiquitous in almost all engineering fields. Control engineering cannot escape this trend. In this paper, we explore how AI tools can be useful in control applications. The core tool we focus on is automatic differentiation. Two immediate applications are linearization of system dynamics for local stability analysis or for state estimation using Kalman filters. We also explore other usages such as conversion of differential algebraic equations to ordinary differential equations for control design. In addition, we explore the use of machine learning models for global parameterizations of state vectors and control inputs in model predictive control applications. For each considered use case, we give examples and results.
Optimal control schemes have achieved remarkable performance in numerous engineering applications. However, they typically require high computational cost, which has limited their use in real-world engineering systems with fast dynamics and/or limited computation power. To address this challenge, Neighboring Extremal (NE) has been developed as an efficient optimal adaption strategy to adapt a pre-computed nominal control solution to perturbations from the nominal trajectory. The resulting control law is a time-varying feedback gain that can be pre-computed along with the original optimal control problem, and it takes negligible online computation. However, existing NE frameworks only deal with state perturbations while in modern applications, optimal controllers (e.g., predictive controllers) frequently incorporate preview information. Therefore, a new NE framework is needed to adapt to such preview perturbations. In this work, an extended NE (ENE) framework is developed to systematically adapt the nominal control to both state and preview perturbations. We show that the derived ENE law is two time-varying feedback gains on the state perturbation and the preview perturbation. We also develop schemes to handle nominal non-optimal solutions and large perturbations to retain optimal performance and constraint satisfaction. Case study on nonlinear model predictive control is presented due to its popularity but it can be easily extended to other optimal control schemes. Promising simulation results on the cart inverted pendulum problem demonstrate the efficacy of the ENE algorithm.
Recent work in imitation learning has shown that having an expert controller that is both suitably smooth and stable enables stronger guarantees on the performance of the learned controller. However, constructing such smoothed expert controllers for arbitrary systems remains challenging, especially in the presence of input and state constraints. As our primary contribution, we show how such a smoothed expert can be designed for a general class of systems using a log-barrier-based relaxation of a standard Model Predictive Control (MPC) optimization problem. At the crux of this theoretical guarantee on smoothness is a new lower bound we prove on the optimality gap of the analytic center associated with a convex Lipschitz function, which we hope could be of independent interest. We validate our theoretical findings via experiments, demonstrating the merits of our smoothing approach over randomized smoothing.
Model predictive control (MPC) is promising for fueling and core density feedback control in nuclear fusion tokamaks, where the primary actuators, frozen hydrogen fuel pellets fired into the plasma, are discrete. Previous density feedback control approaches have only approximated pellet injection as a continuous input due to the complexity that it introduces. In this letter, we model plasma density and pellet injection as a hybrid system and propose two MPC strategies for density control: mixed-integer (MI) MPC using a conventional mixed-integer programming (MIP) solver and MPC utilizing our novel modification of the penalty term homotopy (PTH) algorithm. By relaxing the integer requirements, the PTH algorithm transforms the MIP problem into a series of continuous optimization problems, reducing computational complexity. Our novel modification to the PTH algorithm ensures that it can handle path constraints, making it viable for constrained hybrid MPC in general. Both strategies perform well with regards to reference tracking without violating path constraints and satisfy the computation time limit for real-time control of the pellet injection system. However, the computation time of the PTH-based MPC strategy consistently outpaces the conventional MI-MPC strategy.
Model-based reinforcement learning (RL) has shown great promise due to its sample efficiency, but still struggles with long-horizon sparse-reward tasks, especially in offline settings where the agent learns from a fixed dataset. We hypothesize that model-based RL agents struggle in these environments due to a lack of long-term planning capabilities, and that planning in a temporally abstract model of the environment can alleviate this issue. In this paper, we make two key contributions: 1) we introduce an offline model-based RL algorithm, IQL-TD-MPC, that extends the state-of-the-art Temporal Difference Learning for Model Predictive Control (TD-MPC) with Implicit Q-Learning (IQL); 2) we propose to use IQL-TD-MPC as a Manager in a hierarchical setting with any off-the-shelf offline RL algorithm as a Worker. More specifically, we pre-train a temporally abstract IQL-TD-MPC Manager to predict "intent embeddings", which roughly correspond to subgoals, via planning. We empirically show that augmenting state representations with intent embeddings generated by an IQL-TD-MPC manager significantly improves off-the-shelf offline RL agents' performance on some of the most challenging D4RL benchmark tasks. For instance, the offline RL algorithms AWAC, TD3-BC, DT, and CQL all get zero or near-zero normalized evaluation scores on the medium and large antmaze tasks, while our modification gives an average score over 40.
This work introduces a stochastic model predictive control scheme for dynamic chance constraints. We consider linear discrete-time systems affected by unbounded additive stochastic disturbance. To synthesize an optimal controller, we solve two subsequent stochastic optimization problems. The first problem concerns finding the maximal feasible probabilities of the dynamic chance constraints. After obtaining the probabilities, the second problem concerns finding an optimal controller using stochastic model predictive control. We solve both stochastic optimization problems by reformulating them into deterministic ones using probabilistic reachable tubes and constraint tightening. We prove that the developed algorithm is recursively feasible and yields closed-loop satisfaction of the dynamic chance constraints. In addition, we will introduce a novel implementation using zonotopes to describe the tightening analytically. Finally, we will end with an example illustrating the method's benefits.
Model predictive control (MPC) is a popular approach for trajectory optimization in practical robotics applications. MPC policies can optimize trajectory parameters under kinodynamic and safety constraints and provide guarantees on safety, optimality, generalizability, interpretability, and explainability. However, some behaviors are complex and it is difficult to hand-craft an MPC objective function. A special class of MPC policies called Learnable-MPC addresses this difficulty using imitation learning from expert demonstrations. However, they require the demonstrator and the imitator agents to be identical which is hard to satisfy in many real world applications of robotics. In this paper, we address the practical problem of training Learnable-MPC policies when the demonstrator and the imitator do not share the same dynamics and their state spaces may have a partial overlap. We propose a novel approach that uses a generative adversarial network (GAN) to minimize the Jensen-Shannon divergence between the state-trajectory distributions of the demonstrator and the imitator. We evaluate our approach on a variety of simulated robotics tasks of DeepMind Control suite and demonstrate the efficacy of our approach at learning the demonstrator's behavior without having to copy their actions.
Lately, Nonlinear Model Predictive Control (NMPC)has been successfully applied to (semi-) autonomous driving problems and has proven to be a very promising technique. However, accurate control models for real vehicles could require costly and time-demanding specific measurements. To address this problem, the exploitation of system data to complement or derive the prediction model of the NMPC has been explored, employing learning dynamics approaches within Learning-based NMPC (LbNMPC). Its application to the automotive field has focused on discrete grey-box modeling, in which a nominal dynamics model is enhanced by the data-driven component. In this manuscript, we present an LbNMPC controller for a real go-kart based on a continuous black-box model of the accelerations obtained by Gaussian Processes. We show the effectiveness of the proposed approach by testing the controller on a real go-kart vehicle, highlighting the approximation steps required to get an exploitable GP model on a real-time application.