Integer-order calculus often falls short in capturing the long-range dependencies and memory effects found in many real-world processes. Fractional calculus addresses these gaps via fractional-order integrals and derivatives, but fractional-order dynamical systems pose substantial challenges in system identification and optimal control due to the lack of standard control methodologies. In this paper, we theoretically derive the optimal control via linear quadratic regulator (LQR) for fractional-order linear time-invariant (FOLTI) systems and develop an end-to-end deep learning framework based on this theoretical foundation. Our approach establishes a rigorous mathematical model, derives analytical solutions, and incorporates deep learning to achieve data-driven optimal control of FOLTI systems. Our key contributions include: (i) proposing an innovative system identification method control strategy for FOLTI systems, (ii) developing the first end-to-end data-driven learning framework, Fractional-Order Learning for Optimal Control (FOLOC), that learns control policies from observed trajectories, and (iii) deriving a theoretical analysis of sample complexity to quantify the number of samples required for accurate optimal control in complex real-world problems. Experimental results indicate that our method accurately approximates fractional-order system behaviors without relying on Gaussian noise assumptions, pointing to promising avenues for advanced optimal control.
Nonlinear optimal control is vital for numerous applications but remains challenging for unknown systems due to the difficulties in accurately modelling dynamics and handling computational demands, particularly in high-dimensional settings. This work develops a theoretically certifiable framework that integrates a modified Koopman operator approach with model-based reinforcement learning to address these challenges. By relaxing the requirements on observable functions, our method incorporates nonlinear terms involving both states and control inputs, significantly enhancing system identification accuracy. Moreover, by leveraging the power of neural networks to solve partial differential equations (PDEs), our approach is able to achieving stabilizing control for high-dimensional dynamical systems, up to 9-dimensional. The learned value function and control laws are proven to converge to those of the true system at each iteration. Additionally, the accumulated cost of the learned control closely approximates that of the true system, with errors ranging from $10^{-5}$ to $10^{-3}$.
This paper presents a one-shot learning approach with performance and robustness guarantees for the linear quadratic regulator (LQR) control of stochastic linear systems. Even though data-based LQR control has been widely considered, existing results suffer either from data hungriness due to the inherently iterative nature of the optimization formulation (e.g., value learning or policy gradient reinforcement learning algorithms) or from a lack of robustness guarantees in one-shot non-iterative algorithms. To avoid data hungriness while ensuing robustness guarantees, an adaptive dynamic programming formalization of the LQR is presented that relies on solving a Bellman inequality. The control gain and the value function are directly learned by using a control-oriented approach that characterizes the closed-loop system using data and a decision variable from which the control is obtained. This closed-loop characterization is noise-dependent. The effect of the closed-loop system noise on the Bellman inequality is considered to ensure both robust stability and suboptimal performance despite ignoring the measurement noise. To ensure robust stability, it is shown that this system characterization leads to a closed-loop system with multiplicative and additive noise, enabling the application of distributional robust control techniques. The analysis of the suboptimality gap reveals that robustness can be achieved without the need for regularization or parameter tuning. The simulation results on the active car suspension problem demonstrate the superiority of the proposed method in terms of robustness and performance gap compared to existing methods.
Data-enabled predictive control (DeePC) has garnered significant attention for its ability to achieve safe, data-driven optimal control without relying on explicit system models. Traditional DeePC methods use pre-collected input/output (I/O) data to construct Hankel matrices for online predictive control. However, in systems with evolving dynamics or insufficient pre-collected data, incorporating real-time data into the DeePC framework becomes crucial to enhance control performance. This paper proposes an online DeePC framework for time-varying systems (i.e., systems with evolving dynamics), enabling the algorithm to update the Hankel matrix online by adding real-time informative signals. By exploiting the minimum non-zero singular value of the Hankel matrix, the developed online DeePC selectively integrates informative data and effectively captures evolving system dynamics. Additionally, a numerical singular value decomposition technique is introduced to reduce the computational complexity for updating a reduced-order Hankel matrix. Simulation results on two cases, a linear time-varying system and the vehicle anti-rollover control, demonstrate the effectiveness of the proposed online reduced-order DeePC framework.
The linear programming (LP) approach is, together with value iteration and policy iteration, one of the three fundamental methods to solve optimal control problems in a dynamic programming setting. Despite its simple formulation, versatility, and predisposition to be employed in model-free settings, the LP approach has not enjoyed the same popularity as the other methods. The reason is the often poor scalability of the exact LP approach and the difficulty to obtain bounded solutions for a reasonable amount of constraints. We mitigate these issues here, by investigating fundamental geometric features of the LP and developing sufficient conditions to guarantee finite solutions with minimal constraints. In the model-free context, we show that boundedness can be guaranteed by a suitable choice of dataset and objective function.
To avoid complex constraints of the traditional nonlinear method for tethered space robot (TSR) deployment, this paper proposes a data-driven optimal control framework with an improved deep learning based Koopman operator that could be applied to complex environments. In consideration of TSR's nonlinearity, its finite dimensional lifted representation is derived with the state-dependent only embedding functions in the Koopman framework. A deep learning approach is adopted to approximate the global linear representation of TSR. Deep neural networks (DNN) are developed to parameterize Koopman operator and its embedding functions. An auxiliary neural network is developed to encode the nonlinear control term of finite dimensional lifted system. In addition, the state matrix A and control matrix B of lifted linear system in the embedding space are also estimated during training DNN. Then three loss functions that related to reconstruction and prediction ability of network and controllability of lifted linear system are designed for training the entire network. With the global linear system produced from DNN, Linear Quadratic Regulator (LQR) is applied to derive the optimal control policy for the TSR deployment. Finally, simulation results verify the effectiveness of proposed framework and show that it could deploy tethered space robot more quickly with less swing of in-plane angle.
In this letter, we discuss the problem of optimal control for affine systems in the context of data-driven linear programming. First, we introduce a unified framework for the fixed point characterization of the value function, Q-function and relaxed Bellman operators. Then, in a model-free setting, we show how to synthesize and estimate Bellman inequalities from a small but sufficiently rich dataset. To guarantee exploration richness, we complete the extension of Willem's fundamental lemma to affine systems.
Data-driven control based on the fundamental lemma by Willems et al. is frequently considered for deterministic LTI systems subject to measurement noise. However, besides measurement noise, stochastic disturbances might also directly affect the dynamics. In this paper, we leverage Polynomial Chaos Expansions (PCE) to extend the deterministic fundamental lemma towards stochastic systems. This extension allows to predict future statistical distributions of the inputs and outputs for stochastic LTI systems in data-driven fashion, i.e., based on the knowledge of previously recorded input-output-disturbance data and of the disturbance distribution we perform data-driven uncertainty propagation. Finally, we analyze data-driven stochastic optimal control problems and we propose a conceptual framework for data-driven stochastic predictive control. Numerical examples illustrate the efficacy of the proposed concepts.
Effective postprandial glucose control is important to glucose management for subjects with diabetes mellitus. In this work, a data-driven meal bolus decision method is proposed without the need of subject-specific glucose management parameters. The postprandial glucose dynamics is learnt using Gaussian process regression. Considering the asymmetric risks of hyper- and hypoglycemia and the uncertainties in the predicted glucose trajectories, an asymmetric risk-sensitive cost function is designed. Bayesian optimization is utilized to solve the optimization problem, since the gradient of the cost function is unavailable. The proposed approach is evaluated using the 10-adult cohort of the FDA-accepted UVA/Padova T1DM simulator and compared with the standard insulin bolus calculator. For the case of announced meals, the proposed method achieves satisfactory and similar performance in terms of mean glucose and percentage time in [70, 180] mg/dL without increasing the risk of hypoglycemia. Similar results are observed for the case without the meal information (assuming that the patient follows a consistent diet) and the case of basal rate mismatches. In addition, advisory-mode analysis is performed based on clinical data, which indicates that the method can determine safe and reasonable meal boluses in real clinical settings. The results verify the effectiveness and robustness of the proposed method and indicate the feasibility of achieving improved postprandial glucose regulation through a data-driven optimal control method.