LQR Learning Pipelines

Transcript

1. Data-Driven Control Learning Pipelines general perspectives & recent results for LQR Florian Dörfler SNU Seminar, October 25, 2024
2. general perspectives on an exploding field
3. 3 Increasing role of data-centric methods in science / engineering / industry due to • methodological advances in statistics, optimization, & machine learning • unprecedented availability of brute force • & frenzy surrounding machine learning Works too well to be ignored! Also in control? Thoughts on data in control systems
4. 4 Scientific landscape long & rich history (auto-tuning, system identification, adaptive control, RL, …) & vast + fragmented research landscape → useful binary classification: direct vs indirect control (or model-based vs data-driven) → cynical binary classification: methods working in practice vs methods with a theory
5. 5 Fundamental lemma Surge of methods based on behavioral systems theory Behavior: a linear system is a subspace of trajectories. → basis also for subspace ID → representation-free: subspace represented by models (ss/tf/arx)… or data
6. 6 Bird’s eye view on exploding theory literature today’s angle ICCAS plenary next week
7. 7 Methods also work unreasonably well in practice two weeks ago: DeePC on 100kW inverter connected to the European power grid
8. 8 Data-driven pipelines • indirect (model-based) approach: data → model + uncertainty → control • direct (model-free) approach: direct MRAC, RL, behavioral, … ID • episodic & batch algorithms: collect batch of data → design policy • online & adaptive algorithms: measure → update policy → actuate well-documented trade-offs concerning • complexity: data, compute, & analysis • goal: optimality vs (robust) stability • practicality: modular vs end-to-end … → gold(?) standard: direct, adaptive, optimal yet robust, cheap, & tractable
9. narrow it down to a tractable case study
10. 10 Back to basics: LQR • cornerstone of automatic control • parameterization (can be posed as convex SDP, as differentiable program, as… ) • the benchmark for all data-driven control approaches in last decades but there is no direct & adaptive LQR x+ = Ax + Bu + d z = Q1/ 2x + R1/ 2u K x d u = K x z indirect direct online adaptive offline batch
11. 11 Contents 1. model-based pipeline with model-free elements → data-driven parametrization & robustifying regularization 2. model-free pipeline with model-based elements → adaptive method: policy gradient & sample covariance 3. case studies: academic & power systems/electronics → LQR is academic example but can be made useful
12. 12 Contents 1. regularizations bridging direct & indirect data-driven LQR → story of a model-based pipeline with model-free elements with Pietro Tesi (Florence) & Claudio de Persis (Groningen)
13. 13 are identification, and certainty-equivalence control The conventional approach to data-driven LQR is indirect: t a parametric state-space model is identified from data, d later on controllers are synthesized based on this model n Section II-A. We will briefly review this approach. Regarding the identification task, consider a T-long time es of inputs, disturbances, states, and successor states U0 := ⇥ u(0) u(1) . . . u(T − 1) ⇤ 2 Rm ⇥T , D0 := ⇥ d(0) d(1) . . . d(T − 1) ⇤ 2 Rn⇥T , X 0 := ⇥ x(0) x(1) . . . x(T − 1) ⇤ 2 Rn⇥T , X 1 := ⇥ x(1) x(2) . . . x(T) ⇤ 2 Rn⇥T sfying the dynamics (1), that is, X 1 − D0 = ⇥ B A ⇤ U0 X 0 . (5) s convenient to record the data as consecutive time series, , column i of X 1 coincides with column i + 1 of X 0 , but s is not strictly needed for our developments: the data may ginate from independent experiments. Let for brevity are identification, and certainty-equivalence control The conventional approach to data-driven LQR is indirect: t a parametric state-space model is identified from data, d later on controllers are synthesized based on this model n Section II-A. We will briefly review this approach. Regarding the identification task, consider a T-long time es of inputs, disturbances, states, and successor states U0 := ⇥ u(0) u(1) . . . u(T − 1) ⇤ 2 Rm ⇥T , D0 := ⇥ d(0) d(1) . . . d(T − 1) ⇤ 2 Rn⇥T , X 0 := ⇥ x(0) x(1) . . . x(T − 1) ⇤ 2 Rn⇥T , X 1 := ⇥ x(1) x(2) . . . x(T) ⇤ 2 Rn⇥T sfying the dynamics (1), that is, X 1 − D0 = ⇥ B A ⇤ U0 X 0 . (5) s convenient to record the data as consecutive time series, , column i of X 1 coincides with column i + 1 of X 0 , but s is not strictly needed for our developments: the data may ginate from independent experiments. Let for brevity > > : z(k) = Q 0 0 R1/ 2 x(k) u(k) where k 2 N, x 2 Rn is the state, u 2 Rm is the control input, d isadisturbanceterm, and z istheperformancesignal of interest. We assume that (A, B) is stabilizable. Finally, Q 0 and R 0 are weighting matrices. Here, (⌫ ) and ≺ ( ) denote positive and negative (semi)definiteness. The problem of interest is linear quadratic regulation phrased as designing a state-feedback gain K that renders A + BK Schur and minimizes the H2 -norm of the transfer function T (K ) := d ! z of the closed-loop system1 x(k + 1) z(k) = 2 4 A + BK I Q1/ 2 R1/ 2K 0 3 5 x(k) d(k) , (2) where our notation T (K ) emphasizes the dependence of the transfer function on K . When A + BK is Schur, it holds that kT (K )k2 2 = trace(QP) + trace K > RK P , (3) where P is the controllability Gramian of the closed-loop system (2), which coincides with the unique solution to the > first a parametric state-space model is iden and later on controllers are synthesized bas as in Section II-A. We will briefly review t Regarding the identification task, conside series of inputs, disturbances, states, and su U0 := ⇥ u(0) u(1) . . . u(T − 1) ⇤ 2 D0 := ⇥ d(0) d(1) . . . d(T − 1) ⇤ 2 X 0 := ⇥ x(0) x(1) . . . x(T − 1) ⇤ 2 X 1 := ⇥ x(1) x(2) . . . x(T) ⇤ 2 Rn satisfying the dynamics (1), that is, X 1 − D0 = ⇥ B A ⇤ U0 X 0 It is convenient to record the data as consec i.e., column i of X 1 coincides with column this is not strictly needed for our developme originate from independent experiments. Le W0 := U0 X . > > : z(k) = Q 0 0 R1/ 2 x(k) u(k) where k 2 N, x 2 Rn is the state, u 2 Rm is the control input, d isadisturbanceterm, and z istheperformancesignal of interest. We assume that (A, B) is stabilizable. Finally, Q 0 and R 0 are weighting matrices. Here, (⌫ ) and ≺ ( ) denote positive and negative (semi)definiteness. The problem of interest is linear quadratic regulation phrased as designing a state-feedback gain K that renders A + BK Schur and minimizes the H2 -norm of the transfer function T (K ) := d ! z of the closed-loop system1 x(k + 1) z(k) = 2 4 A + BK I Q1/ 2 R1/ 2K 0 3 5 x(k) d(k) , (2) where our notation T (K ) emphasizes the dependence of the transfer function on K . When A + BK is Schur, it holds that kT (K )k2 2 = trace(QP) + trace K > RK P , (3) where P is the controllability Gramian of the closed-loop system (2), which coincides with the unique solution to the first a parametric state-space model is iden and later on controllers are synthesized base as in Section II-A. We will briefly review t Regarding the identification task, conside series of inputs, disturbances, states, and su U0 := ⇥ u(0) u(1) . . . u(T − 1) ⇤ 2 D0 := ⇥ d(0) d(1) . . . d(T − 1) ⇤ 2 X 0 := ⇥ x(0) x(1) . . . x(T − 1) ⇤ 2 X 1 := ⇥ x(1) x(2) . . . x(T) ⇤ 2 Rn satisfying the dynamics (1), that is, X 1 − D0 = ⇥ B A ⇤ U0 X 0 It is convenient to record the data as consec i.e., column i of X 1 coincides with column this is not strictly needed for our developme originate from independent experiments. Le W0 := U0 . X 1 = AX 0 + BU0 + D0 Indirect & certainty-equivalence LQR • collect I/O data (𝑋0 , 𝑈0 , 𝑋1 ) with 𝐷0 unknown & PE: rank 𝑈0 𝑋0 = 𝑛 + 𝑚 • indirect & certainty- equivalence LQR (optimal in MLE setting) least squares SysID certainty- equivalent LQR
14. 14 are identification, and certainty-equivalence control The conventional approach to data-driven LQR is indirect: t a parametric state-space model is identified from data, d later on controllers are synthesized based on this model n Section II-A. We will briefly review this approach. Regarding the identification task, consider a T-long time es of inputs, disturbances, states, and successor states U0 := ⇥ u(0) u(1) . . . u(T − 1) ⇤ 2 Rm ⇥T , D0 := ⇥ d(0) d(1) . . . d(T − 1) ⇤ 2 Rn⇥T , X 0 := ⇥ x(0) x(1) . . . x(T − 1) ⇤ 2 Rn⇥T , X 1 := ⇥ x(1) x(2) . . . x(T) ⇤ 2 Rn⇥T sfying the dynamics (1), that is, X 1 − D0 = ⇥ B A ⇤ U0 X 0 . (5) s convenient to record the data as consecutive time series, , column i of X 1 coincides with column i + 1 of X 0 , but s is not strictly needed for our developments: the data may ginate from independent experiments. Let for brevity are identification, and certainty-equivalence control The conventional approach to data-driven LQR is indirect: t a parametric state-space model is identified from data, d later on controllers are synthesized based on this model n Section II-A. We will briefly review this approach. Regarding the identification task, consider a T-long time es of inputs, disturbances, states, and successor states U0 := ⇥ u(0) u(1) . . . u(T − 1) ⇤ 2 Rm ⇥T , D0 := ⇥ d(0) d(1) . . . d(T − 1) ⇤ 2 Rn⇥T , X 0 := ⇥ x(0) x(1) . . . x(T − 1) ⇤ 2 Rn⇥T , X 1 := ⇥ x(1) x(2) . . . x(T) ⇤ 2 Rn⇥T sfying the dynamics (1), that is, X 1 − D0 = ⇥ B A ⇤ U0 X 0 . (5) s convenient to record the data as consecutive time series, , column i of X 1 coincides with column i + 1 of X 0 , but s is not strictly needed for our developments: the data may ginate from independent experiments. Let for brevity > > : z(k) = Q 0 0 R1/ 2 x(k) u(k) where k 2 N, x 2 Rn is the state, u 2 Rm is the control input, d isadisturbanceterm, and z istheperformancesignal of interest. We assume that (A, B) is stabilizable. Finally, Q 0 and R 0 are weighting matrices. Here, (⌫ ) and ≺ ( ) denote positive and negative (semi)definiteness. The problem of interest is linear quadratic regulation phrased as designing a state-feedback gain K that renders A + BK Schur and minimizes the H2 -norm of the transfer function T (K ) := d ! z of the closed-loop system1 x(k + 1) z(k) = 2 4 A + BK I Q1/ 2 R1/ 2K 0 3 5 x(k) d(k) , (2) where our notation T (K ) emphasizes the dependence of the transfer function on K . When A + BK is Schur, it holds that kT (K )k2 2 = trace(QP) + trace K > RK P , (3) where P is the controllability Gramian of the closed-loop system (2), which coincides with the unique solution to the > first a parametric state-space model is iden and later on controllers are synthesized bas as in Section II-A. We will briefly review t Regarding the identification task, conside series of inputs, disturbances, states, and su U0 := ⇥ u(0) u(1) . . . u(T − 1) ⇤ 2 D0 := ⇥ d(0) d(1) . . . d(T − 1) ⇤ 2 X 0 := ⇥ x(0) x(1) . . . x(T − 1) ⇤ 2 X 1 := ⇥ x(1) x(2) . . . x(T) ⇤ 2 Rn satisfying the dynamics (1), that is, X 1 − D0 = ⇥ B A ⇤ U0 X 0 It is convenient to record the data as consec i.e., column i of X 1 coincides with column this is not strictly needed for our developme originate from independent experiments. Le W0 := U0 X . > > : z(k) = Q 0 0 R1/ 2 x(k) u(k) where k 2 N, x 2 Rn is the state, u 2 Rm is the control input, d isadisturbanceterm, and z istheperformancesignal of interest. We assume that (A, B) is stabilizable. Finally, Q 0 and R 0 are weighting matrices. Here, (⌫ ) and ≺ ( ) denote positive and negative (semi)definiteness. The problem of interest is linear quadratic regulation phrased as designing a state-feedback gain K that renders A + BK Schur and minimizes the H2 -norm of the transfer function T (K ) := d ! z of the closed-loop system1 x(k + 1) z(k) = 2 4 A + BK I Q1/ 2 R1/ 2K 0 3 5 x(k) d(k) , (2) where our notation T (K ) emphasizes the dependence of the transfer function on K . When A + BK is Schur, it holds that kT (K )k2 2 = trace(QP) + trace K > RK P , (3) where P is the controllability Gramian of the closed-loop system (2), which coincides with the unique solution to the first a parametric state-space model is iden and later on controllers are synthesized base as in Section II-A. We will briefly review t Regarding the identification task, conside series of inputs, disturbances, states, and su U0 := ⇥ u(0) u(1) . . . u(T − 1) ⇤ 2 D0 := ⇥ d(0) d(1) . . . d(T − 1) ⇤ 2 X 0 := ⇥ x(0) x(1) . . . x(T − 1) ⇤ 2 X 1 := ⇥ x(1) x(2) . . . x(T) ⇤ 2 Rn satisfying the dynamics (1), that is, X 1 − D0 = ⇥ B A ⇤ U0 X 0 It is convenient to record the data as consec i.e., column i of X 1 coincides with column this is not strictly needed for our developme originate from independent experiments. Le W0 := U0 . X 1 = AX 0 + BU0 + D0 Direct approach from subspace relations in data • PE data: rank 𝑈0 𝑋0 = 𝑛 + 𝑚 • subspace relations • data-driven LQR LMIs by substituting → certainty equivalence by neglecting noise :
15. 15 Equivalence: direct + xxx  indirect • direct approach • indirect approach → optimizer has nullspace → orthogonality constraint equivalent constraints:
16. 16 Regularized, certainty-equivalent, & direct LQR • orthogonality constraint lifted to regularizer (equivalent for large) … but may not be robust (?) • interpolates between control & SysID • effect of noise entering data: Lyapunov constraint becomes for robustness should be small → forced by small
17. 17 Performance & robustness certificates realized cost from regularized design with large if exact system matrices A & B were known • SNR (signal-to-noise-ratio) • relative performance metric Certificate for sufficiently large SNR: the optimal control problem is feasible (robustly stabilizing) with relative performance ~ 𝒪 Τ (1 𝑆𝑁𝑅).
18. 18 Numerical case study • case study [Dean et al. ‘19]: discrete-time system with noise variance 𝜎2 = 0.01 & variable regularization coefficient 𝜆 • take-home message: regularization is needed for robustness & performance % of stabilizing controllers (100 trials) median relative performance error breaks without regularizer regularization coefficient 𝜆 → works… but lame: learning is offline
19. 19 Contents 2. data-enabled policy optimization for online adaptation → story of a model-free pipeline with model-based elements with Feiran Zhao (ETH), Alessandro Chiuso (Padova), Linbin Huang (ETH & Zhejiang), & Keyou You (Tsinghua)
20. 20 Online & adaptive solutions • shortcoming of separating offline learning & online control → cannot improve policy online & cheaply / rapidly adapt to changes • (elitist) desired adaptive solution: direct, online (non-episodic/non-batch) algorithms, with closed-loop data, & recursive algorithmic implementation • “best” way to improve policy with new data → go down the gradient ! PII: S0005–1098(98)00089–2 Automatica, Vol. 34, No. 10, pp. 1161— 1167, 1998 1998 IFAC. Published by Elsevier Science Ltd All rights reserved. Printed in Great Britain 0005-1098/98 $—see front matter Adaptive Control: Towards a Complexity-Based General Theory* G. ZAM ES- Key Words—H control; adaptive control; learning control; performance analysis. Abstract—Two recent developments are pointing the way to- wards an input— output theory of H ! l adaptive feedback: The solution of problems involving: (1) feedback performance exact optimization under large plant uncertainty on the one hand (thetwo-disc problem of H ); and (2) optimally fast identi- fication in H on the other. Taken together, these are yielding adaptive algorithms for slowly varying data in H ! l . At a conceptual level, theseresultsmotivatea general input— output theory linking identification, adaptation, and control learning. In such a theory, thedefinition of adaptation isbased on system performance under uncertainty, and is independent of internal structure, presence or absence of variable parameters, or even feedback. 1998 IFAC. Published by Elsevier Science Ltd. All rights reserved. 1. INTRODUCTION certain difficulties. Controllers with identical external behavior can have an endless variety of parametrizations; variable parameters in one parametrization may be replaced by a fixed para- meter nonlinearity in another. In most of therecent control literature there is no clear separation be- tween the concepts of adaptation and nonlinear feedback, or between research on adaptive control and nonlinear stability. This lack of clarity extends to fields other than control; e.g. in debates as to whether neural nets do or do not have a learning capacity;or in theclassical 1960sChomsky vsSkin- ner argument as to whether children’s language “adaptive = improve over best control with a priori info” * disclaimer: a large part of the adaptive control community focuses on stability & not optimality
21. 21 Ingredient 1: policy gradient methods • LQR viewed as smooth program (many formulations) • 𝐽 𝐾 is not convex … after eliminating (unique) P, denote this as 𝐽 𝐾 Fact: policy gradient descent 𝐾+ = 𝐾 − 𝜂 ∇𝐽 𝐾 initialized from a stabilizing policy converges linearly to 𝐾∗. Annual Review of Control, Robotics , and AutonomousSys tems Toward aT heoretical Foundation of Policy Optimization for Learning Control Policies Bin Hu,1 Kaiqing Zhang,2,3 Na Li,4 Mehran Mesbahi,5 Maryam Fazel,6 and Tamer Ba¸ sar1 1Coordinated Science Laboratory and Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign, Urbana, Illinois, USA; email: binhu7@ illinois.edu, basar1@ illinois.edu 2Laboratory for Information and Decision Systems and Computer Science and Artif cial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA 3Current aff liation: Department of Electrical and Computer Engineering and Institute for Systems Research, University of Maryland, College Park, Maryland, USA; email: kaiqing@ umd.edu 4School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts, USA; email: nali@ seas.harvard.edu 5Department of Aeronautics and Astronautics, University of Washington, Seattle, Washington, USA; email: mesbahi@ uw.edu 6Department of Electrical and Computer Engineering, University of Washington, Seattle, Washington, USA; email: mfazel@ uw.edu Annu. Rev. Control Robot. Auton. Syst. 2023. 6:123–58 T he Annual Review of Control, Robotics , and AutonomousSystemsisonline at control.annualreviews.org https://doi.org/10.1146/annurev-control-042920- 020021 Copyright © 2023 by the author(s). T his work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. See credit lines of images or other third-party material in this article for license information. Keywords policy optimization, reinforcement learning, feedback control synthesis Abstract Gradient-based methodshave been widely used for system design and opti- mization in diverse application domains. Recently, there hasbeen arenewed interest in studying theoretical propertiesof thesemethodsin thecontext of control and reinforcement learning. T hisarticle surveyssome of the recent developments on policy optimization, a gradient-based iterative approach for feedback control synthesis that hasbeen popularized by successesof re- inforcement learning. We take an interdisciplinary perspective in our expo- sition that connects control theory, reinforcement learning, and large-scale optimization. We review anumber of recently developed theoretical results on the optimization landscape, global convergence, and sample complexity 123 but on the set of stabilizing gains K , it’s • coercive with compact sublevel sets, • smooth with bounded Hessian, & • degree-2 gradient dominated 𝐽 𝐾 − 𝐽∗ ≤ 𝑐𝑜𝑛𝑠𝑡. ∙ ∇𝐽 𝐾 2
22. 22 Model-free policy gradient methods • policy gradient: 𝐾+ = 𝐾 − 𝜂 ∇𝐽 𝐾 converges linearly to 𝐾∗ • model-based setting: explicit formulae for ∇𝐽 𝐾 based on closed-loop controllability + observability Gramians [Levine & Athans, ‘70] • model-free 0th order methods constructing two-point gradient estimate from numerous & very long trajectories → extremely sample inefficient • IMHO: policy gradient is potentially great candidate for direct adaptive control but sadly useless in practice: sample-inefficient, episodic, … relative performance gap 𝜖 = 1 𝜖 = 0.1 𝜖 = 0.01 # trajectories (100 samples) 1414 43850 142865 ~ 𝟏𝟎𝟕 samples
23. 23 Ingredient 2: sample covariance parameterization prior parameterization • PE condition: full row rank 𝑈0 𝑋0 • 𝐴 + 𝐵𝐾 = 𝐵 𝐴 𝐾 𝐼 = 𝐵 𝐴 𝑈0 𝑋0 𝐺 = 𝑋1 𝐺 • robustness: 𝐺 = 𝑈0 𝑋0 ⊤ ∙ regularization • dimension of all matrices grows with 𝑡 covariance parameterization • sample covariance Λ = 1 𝑡 𝑈0 𝑋0 𝑈0 𝑋0 ⊤ ≻ 0 • 𝐴 + 𝐵𝐾 = 𝐵 𝐴 𝐾 𝐼 = 𝐵 𝐴 Λ𝑉 = 1 𝑡 𝑋1 𝑈0 𝑋0 ⊤ 𝑉 • robustness for free without regularization • dimension of all matrices is constant + cheap rank-1 updates for online data X 1 = AX 0 + BU0 U0 = ⇥ u(0) u(1) ··· u(t − 1) ⇤ X 1 = ⇥ x(1) x(2) ··· x(t) ⇤ X 0 = ⇥ x(0) x(1) ··· x(t − 1) ⇤
24. 24 Covariance parameterization of the LQR • state / input sample covariance Λ = 1 𝑡 𝑈0 𝑋0 𝑈0 𝑋0 ⊤ & 𝑋1 = 1 𝑡 𝑋1 𝑈0 𝑋0 ⊤ • closed-loop matrix 𝐴 + 𝐵𝐾 = 𝑋1 𝑉 with 𝐾 −−−− 𝐼 = Λ 𝑉 = 𝑈0 −−−− 𝑋0 𝑉 • LQR covariance parameterization after eliminating 𝐾 with variable 𝑉, Lyapunov eqn (explicitly solvable), smooth cost 𝐽(𝑉) (after removing 𝑃), & linear parameterization constraint min 𝑉,𝑃≻0 trace 𝑄𝑃 + trace 𝑉𝑇𝑈0 𝑇 𝑅𝑈0 𝑉𝑃 s. t. 𝑃 = 𝐼 + 𝑋1 𝑉 𝑃𝑉𝑇𝑋1 𝑇 , 𝐼 = 𝑋0 𝑉
25. 25 Projected policy gradient with sample covariances • data-enabled policy optimization (DeePO) Π 𝑋0 projects on parameterization constraint 𝐼 = 𝑋0 𝑉 & gradient ∇𝐽 𝑉 is computed from two Lyapunov equations with sample covariances • optimization landscape: smooth, degree-1 proj. grad dominance 𝐽 𝑉 − 𝐽∗ ≤ 𝑐𝑜𝑛𝑠𝑡. ∙ Π 𝑋0 ∇𝐽 𝑉 • warm-up: offline data & no disturbance 𝑉+ = 𝑉 − 𝜂 Π 𝑋0 (∇𝐽 𝑉 ) Sublinear convergence for feasible initialization 𝐽 𝑉𝑘 − 𝐽∗ ≤ 𝒪(1/𝑘) . 𝐽 𝑉𝑘 − 𝐽∗ 𝐽∗ note: empirically faster linear rate case: 4th order system with 8 data samples
26. 26 Online, adaptive, & closed-loop DeePO where 𝑋0,𝑡+1 = 𝑥 0 , 𝑥 1 , … 𝑥 𝑡 , 𝑥(𝑡 + 1) & similar for other matrices • cheap & recursive implementation: rank-1 update of (inverse) sample covariances, cheap computation, & no memory needed to store old data 𝑥+ = 𝐴𝑥 + 𝐵𝑢 + 𝑑 𝑥 𝑢 𝑢 = 𝐾𝑡+1 𝑥 ① update sample covariances: Λ𝑡+1 & ᪄ 𝑋0,𝑡+1 ② update decision variable: 𝑉𝑡+1 = Λ𝑡+1 −1 𝐾𝑡 𝐼𝑛 ③ gradient descent: 𝑉𝑡+1 ′ = 𝑉𝑡+1 − 𝜂Π ᪄ 𝑋0,𝑡+1 (∇𝐽𝑡+1 𝑉𝑡+1 ) ④ update control gain: 𝐾𝑡+1 = ഥ 𝑈0,𝑡+1 𝑉𝑡+1 ′ DeePO policy update Input: (𝑋0,𝑡+1 , 𝑈0,𝑡+1 , 𝑋1,𝑡+1 ), 𝐾𝑡 Output: 𝐾𝑡+1 𝑑 𝐾𝑡+1
27. 27 with Niklas Persson & Alessandro Papadopoulos (Mälardalen University) Eye candy: it works on complex systems
28. 28 Underlying assumptions for theoretic certificates • initially stabilizing controller: the LQR problem parameterized by offline data 𝑋0,𝑡0 , 𝑈0,𝑡0 , 𝑋1,𝑡0 is feasible with stabilizing gain 𝐾𝑡0 . • persistency of excitation due to probing with constant variance 𝛾2 : 𝜎 ℋ𝑛+1 𝑈0,𝑡 ≥ 𝛾 ∙ 𝑡 with Hankel matrix ℋ𝑛+1 𝑈0,𝑡 • bounded noise: 𝑑(𝑡) ≤ 𝛿 ∀ 𝑡 → signal-to-noise ratio 𝑆𝑁𝑅 ≔ Τ 𝛾 𝛿 • BIBO: there are ത 𝑢, ҧ 𝑥 such that 𝑢(𝑡) ≤ ത 𝑢 & 𝑥 𝑡 ≤ ҧ 𝑥 (assumption can be avoided by picking sufficiently small step size)
29. 29 Bounded regret of DeePO in adaptive setting • average regret performance metric Regret𝑇 ≔ 1 𝑇 σ 𝑡=𝑡0 𝑡0+𝑇−1 𝐽 𝐾𝑡 − 𝐽∗ • comments on the qualitatively expected result: • analysis is independent of the noise statistics & consistent Regret𝑇→∞ → 0 • favorable sample complexity: sublinear decrease term matches best rate 𝒪(1/ 𝑇) of first-order methods in online convex optimization • empirically observe smaller bias term: 𝒪( Τ 1 𝑆𝑁𝑅2) & not Τ 𝒪(1 𝑆𝑁𝑅) Sublinear regret: under the assumptions, for sufficiently small step size 𝜂, & for sufficiently large 𝑆𝑁𝑅, it holds that 𝐾𝑡 is stabilizing & Regret𝑇 ≤ 𝑐𝑜𝑛𝑠𝑡. 𝑇 + 𝑐𝑜𝑛𝑠𝑡. 𝑆𝑁𝑅 .
30. 30 Comparison case studies • same case study [Dean et al. ’19] 𝐽 𝐾𝑡 − 𝐽∗ 𝐽∗ • case 1: offline LQR vs direct adaptive DeePO vs indirect adaptive: rls + dlqr → adaptive outperforms offline → direct/indirect rates matching but direct is much(!) cheaper • case 2: adaptive DeePO vs 0𝑡ℎ order methods relative performance gap 𝜖 = 1 𝜖 = 0.1 𝜖 = 0.01 # long trajectories (100 samples) for 0𝑡ℎ order LQR 1414 43850 142865 DeePO (# I/O samples) 10 24 48 → significantly less data
31. 31 Power systems / electronics case study • wind turbine becomes unstable in weak grids with nonlinear oscillations • converter, turbine, & grid are a black box for the commissioning engineer • construct state from time shifts (5ms sampling) of 𝑦 𝑡 , 𝑢(𝑡) & use DeePO synchronous generator & full-scale converter
32. 32 Power systems / electronics case study 0 2 4 6 8 10 12 time [s] (a) 0.84 0.86 0.88 0.9 0.92 0.94 0.96 active power (p.u.) probe & collect data oscillation observed activate DeePO without DeePO with DeePO (100 iterations) with DeePO (1 iteration)
33. 33 … same in the adaptive setting with excitation 0 2 4 6 8 10 12 time [s] (a) 0.84 0.86 0.88 0.9 0.92 0.94 0.96 active power (p.u.) without DeePO with adaptive DeePO probe & collect data oscillation observed activate DeePO
34. 34 Conclusions • Challenge: adaptive data-driven control • model-based pipeline with model-free block: data-driven LQR parametrization → works well when regularized (note: further flexible regularizations available) • model-free pipeline with model-based block: policy gradient & sample covariance → DeePO is adaptive, online, with closed-loop data, & recursive implementation • academic case studies & can be made useful in bikes + power systems/electronics • Future work • technicalities: weaken assumptions & improve rates • control: based on output feedback & for other objectives • complex system classes: stochastic, time-varying, & nonlinear • open questions: online vs episodic? “best” batch size? triggered?
35. 35 Related papers 2. model-free pipeline with model-based elements 1. model-based pipeline with model-free elements
36. 36 Perspective papers
37. 37 thanks