EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

EE 581: Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Department of Electrical Engineering Pennsylvania State University, University Park, USA Instructor: Ji-Woong Lee

April 9, 2009

VII. Pontryagin’s Minimum Principle EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

1 Calculus of Variations

Fixed End Point Problems Examples

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

2 Pontryagin’s Minimum Principle

Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

VII. Pontryagin’s Minimum Principle EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

1 Calculus of Variations

Fixed End Point Problems Examples

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

2 Pontryagin’s Minimum Principle

Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Calculus of Variations Fixed End Point Problems EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

A Normed Vector Space Continuously Differentiable Functions: V = C1n [t0 , t1 ] Norm on V : Given any norm k · k on Rn , kxkV = maxt∈[t0 ,t1 ] kx(t)k + maxt∈[t0 ,t1 ] kx(t)k, ˙ x ∈V Fixed End Point Problem Minimize J(x) over x ∈ V subject to x ∈ Ω. Rt Cost Function: J(x) = t01 F (t, x(t), x(t)) ˙ dt, where F (t, x, y ) is smooth in (t, x, y ) ∈ [t0 , t1 ] × Rn × Rn . Constraint Set: Ω = {x ∈ V : x(t0 ) = x0 , x(t1 ) = x1 }

Example: Find shortest path between (t0 , x0 ) and (t1 , x1 ) p 2 ⇒ F (t, x, y ) = 1 + y

Calculus of Variations Fixed End Point Problems EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Lemma For fixed-end-point calculus of variations problems, Fr´echet Derivative of J: For x ∗ ∈ Ω and x, h ∈ V , ∂J ∗ ∂x (x )h

=

R t1  ∂F t0

˙ + ∂x (t, x(t), x(t))h(t)

∂F ∂y

 ˙ (t, x(t), x˙ (t))h(t) dt.

Tangent Cone to Ω: TC (x ∗ , Ω) = {h ∈ V : h(t0 ) = h(t1 ) = 0} for x ∗ ∈ Ω. Necessary Condition for Optimality: ∂J ∗ ∂x (x )h = 0 for h ∈ V with h(t0 ) = h(t1 ) = 0. Theorem (Euler-Lagrange Equation)

If x ∗ ∈ C2n [t0 , t1 ] is a local minimizer of J on Ω, then ∂F d ∂F (t, x ∗ (t), x˙ ∗ (t)) − (t, x ∗ (t), x˙ ∗ (t)) = 0 for t ∈ [t0 , t1 ]. ∂x dt ∂y

VII. Pontryagin’s Minimum Principle EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

1 Calculus of Variations

Fixed End Point Problems Examples

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

2 Pontryagin’s Minimum Principle

Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Calculus of Variations Examples EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Shortest path between (t0 , x0 ) and (t1 , x1 ): p F (t, x, y ) = 1 + y 2 ∂F 2 −1/2 ⇒ ∂F ∂x (t, x, y ) = 0, ∂y (t, x, y ) = y (1 + y ) E-L Equation: x¨(t) = 0 ⇒ x(t) = c1 t + c2 (straight line) RT Minimum Energy Control: Minimize 0 u(t)2 dt subject to x˙ = ax + u, x(0) = x0 , x(T ) = x1 . (a 6= 0) 2 F (t, x, y ) = (y − ax) ∂F ⇒ ∂F ∂x (t, x, y ) = −2a(y − ax), ∂y (t, x, y ) = 2(y − ax), d ∂F x (t) − ax(t)) ˙ dt ∂y (t, x(t), x˙ (t)) = 2(¨ E-L Equation: x¨(t) − a2 x(t) = 0 ⇒ x(t) = c1 e at + c2 e −at ⇒ x(t) = ⇒ u(t) = =

aT x −x x1 −e −aT x0 at 0 1 −at e + eeaT −e −aT e e aT −e −aT 2a(e aT x0 −x1 ) −at e e aT −e −aT 2ae 2a(T −t) − e 2a(T −t) −1 (x(t) − e −a(T −t) x1 )

(open-loop) (closed-loop)

VII. Pontryagin’s Minimum Principle EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

1 Calculus of Variations

Fixed End Point Problems Examples

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

2 Pontryagin’s Minimum Principle

Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Pontryagin’s Minimum Principle Minimum Terminal Cost Problems EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Nonlinear Systems Nonlinear System: x(t) ˙ = f (t, x(t), u(t)), t ∈ [t0 , tf ] Regularity Conditions: For each t ∈ [t0 , tf ], function f (t, ·, ·) : Rn × Rm → Rn is continuously Fr´echet differentiable. There exists a finite D ⊂ [t0 , tf ] s.t. f , ∂f /∂x, and ∂f /∂u are continuous on ([t0 , tf ] \ D) × Rn × Rm . For α > 0, there exist β, γ > 0 s.t. kf (t, x, u)k ≤ β + γkxk for (t, x, u) ∈ [t0 , tf ] × Rn × Rm with kuk ≤ α.

Existence of Unique Solution: Let t1 ∈ [t0 , tf ], z ∈ Rn , and u ∈ PC m [t0 , tf ]. Under regularity conditions, x(t) ˙ = f (t, x(t), u(t)),

x(t1 ) = z

has a unique solution x(t) = φ(t, t1 , z, u), t ∈ [t1 , tf ].

Pontryagin’s Minimum Principle Minimum Terminal Cost Problems (Cont’d) EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Definitions Set of Admissible Controls: Let Ω ⊂ Rm . A control u is admissible if u ∈ U, where U = {u ∈ PC m [t0 , tf ] : u(t) ∈ Ω for all t ∈ [t0 , t1 ]}.

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Set of Researchable States: Given t1 , t2 ∈ [t0 , tf ] and z ∈ Rn , a state x is reachable at time t2 from x(t1 ) = z if x ∈ K (t2 , t1 , z), where K (t2 , t1 , z) = {φ(t2 , t1 , z, u) : u ∈ U }. Minimum Terminal Cost Problem Given continuously Fr´echet differentiable ψ : Rn → R, minimize J(u) = ψ(x(tf )) subject to x(t) ˙ = f (t, x(t), u(t)) and u(t) ∈ Ω for t ∈ [t0 , tf ], x(t0 ) = x0 , and u ∈ PC m [t0 , tf ].

Pontryagin’s Minimum Principle Minimum Terminal Cost Problems (Cont’d) EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Lemma (Strong Perturbation uε of u ∗ ) Let u ∗ ∈ U be optimal, x ∗ be s.t. x˙ ∗ (t) = f (t, x ∗ (t), u ∗ (t)), t ∈ [t0 , tf ], and D ∗ ⊂ [t0 , t1 ] be the finite set on which u ∗ is discontinuous. Let τ ∈ (t0 , tf ) \ (D ∗ ∪ D), v ∈ Ω, and ( v for t ∈ [τ − ε, τ ] and t ≥ t0 ; uε (t) = ∗ u (t) otherwise for ε > 0. Then there exist o(t, ·), t ∈ [t0 , tf ], s.t. xε (t) = x ∗ (t) + εhτ,v (t) + o(t, ε),

t ∈ [t0 , tf ],

with o(t, ε)/ε → 0 for each t ∈ [t0 , tf ] as ε → 0 and hτ,v (t) =

(

0, Φ(t, τ ) [f (τ, x ∗ (τ ), v ) − f (τ, x ∗ (τ ), u ∗ (τ ))] ,

where Φ is the state transition matrix of

t ∈ [t0 , τ ); t ∈ [τ, tf ],

∂f ∗ ∗ ∂x (·, x (·), u (·)).

Pontryagin’s Minimum Principle Minimum Terminal Cost Problems (Cont’d) EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Lemma Let u ∗ ∈ U be optimal, x ∗ be s.t. x˙ ∗ (t) = f (t, x ∗ (t), u ∗ (t)), t ∈ [t0 , tf ], and D ∗ ⊂ [t0 , t1 ] be the finite set on which u ∗ is discontinuous. Then ∇ψ(x ∗ (tf ))T h ≥ 0 for h ∈ co TC (x ∗ (tf ), K (tf , t0 , x0 )) hτ,v (t) ∈ TC (x ∗ (t), K (t, t0 , x0 )), t ∈ [t0 , tf ], whenever τ ∈ (t0 , tf ) \ (D ∗ ∪ D) and v ∈ Ω. Hamiltonian The Hamiltonian H : [t0 , tf ] × Rn × Rm × Rn → R for the minimum terminal cost problem is defined by H(t, x, u, p) = p T f (t, x, u) for (t, x, u, p) ∈ [t0 , tf ] × Rn × Rm × Rn .

Pontryagin’s Minimum Principle Minimum Terminal Cost Problems (Cont’d) EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Pontryagin’s Minimum Principle: Minimum Terminal Cost Suppose u ∗ ∈ U is optimal for the minimum terminal cost problem. Let x ∗ solve the state equation x˙ ∗ (t) = f (t, x ∗ (t), u ∗ (t)),

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

t ∈ [t0 , tf ],

with x ∗ (t0 ) = x0 , and let p ∗ solve the costate equation ∂f (t, x ∗ (t), u ∗ (t))T p ∗ (t), t ∈ [t0 , tf ], ∂x with p ∗ (tf ) = ∇ψ(x ∗ (tf )). Then the minimum principle p˙ ∗ (t) = −

H(t, x ∗ (t), u ∗ (t), p ∗ (t)) = inf H(t, x ∗ (t), v , p ∗ (t)) v ∈Ω

holds for all t ∈ [t0 , tf ] except possibly for a finite set. n m Moreover, if ∂f ∂t (t, x, u) = 0 for (t, x, u) ∈ [t0 , tf ] × R × R , then inf v ∈Ω H(·, x ∗ (·), v , p ∗ (·)) is constant on [t0 , tf ].

VII. Pontryagin’s Minimum Principle EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

1 Calculus of Variations

Fixed End Point Problems Examples

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

2 Pontryagin’s Minimum Principle

Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Pontryagin’s Minimum Principle Free End Point Problems EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Free End Point Problem Let f0 : [t0 , tf ] × Rn × Rm → R and f : [t0 , tf ] × Rn × Rm → Rn satisfy regularity conditions; let ψ : Rn → R be continuously Fr´echet differentiable. Minimize Z tf J(u) = f0 (t, x(t), u(t)) dt + ψ(x(tf )) t0

subject to x(t) ˙ = f (t, x(t), u(t)) and u(t) ∈ Ω for t ∈ [t0 , tf ], x(t0 ) = x0 , and u ∈ PC m [t0 , tf ]. Transformed Problem Minimize J(u) = x0 (tf ) + ψ(x(tf )) subject to         x˙ 0 (t) f0 (t, x(t), u(t)) x0 (t0 ) 0 = , = , etc. x(t) ˙ f (t, x(t), u(t)) x(t0 ) x0

Pontryagin’s Minimum Principle Free End Point Problems (Cont’d) EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Pontryagin’s Minimum Principle: Free End Point Suppose u ∗ ∈ U is optimal for the free end point problem. Let x˙ ∗ (t) = f (t, x ∗ (t), u ∗ (t)), ∂f0 ∂f p˙ ∗ (t) = − (t, x ∗ (t), u ∗ (t))T − (t, x ∗ (t), u ∗ (t))T p ∗ (t) ∂x ∂x for t ∈ [t0 , tf ], with x ∗ (t0 ) = x0 and p ∗ (tf ) = ∇ψ(x ∗ (tf )). For (t, x, u, p) ∈ [t0 , tf ] × Rn × Rm × Rn , define the Hamiltonian H(t, x, u, p) = f0 (t, x, u) + p T f (t, x, u). Then the minimum principle H(t, x ∗ (t), u ∗ (t), p ∗ (t)) = inf v ∈Ω H(t, x ∗ (t), v , p ∗ (t)) holds for all t ∈ [t0 , tf ] except possibly for a finite set. ∂f 0 Moreover, if ∂f ∂t and ∂t are identically zero, then inf v ∈Ω H(·, x ∗ (·), v , p ∗ (·)) is constant on [t0 , tf ].

VII. Pontryagin’s Minimum Principle EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

1 Calculus of Variations

Fixed End Point Problems Examples

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

2 Pontryagin’s Minimum Principle

Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Pontryagin’s Minimum Principle Fixed End Point Problems EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Lemma (Transversality Condition) Consider the minimum terminal cost problem with additional constraint gf (x(tf )) = 0, where gf : Rn → Rmf is continuously Fr´echet differentiable. Let u ∗ ∈ U be optimal, x ∗ (t) = ∗ f φ(t, t0 , x0 , u ∗ ), t ∈ [t0 , tf ], and ∂g ∂x (x (tf )) have full row rank. There exist µ0 ∈ {0, 1} and µ ∈ Rmf s.t. (µ0 , µ) 6= (0, 0)  T ∗ (t ))T µ f and µ0 ∇ψ(x ∗ (tf )) + ∂g (x h ≥ 0 for f ∂x h ∈ co TC (x ∗ (tf ), K (tf , t0 , x0 )).

In Pontryagin’s minimum principle, p ∗ (tf ) = ∇ψ(x ∗ (tf )) is replaced with the following transversality condition: There exist µ0 ∈ {0, 1} and µ ∈ Rmf s.t. (µ0 , µ) 6= (0, 0) and ∂gf ∗ p ∗ (tf ) = µ0 ∇ψ(x ∗ (tf )) + (x (tf ))T µ. ∂x

Pontryagin’s Minimum Principle Fixed End Point Problems (Cont’d) EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Fixed End Point Problem Let f0 : [t0 , tf ] × Rn × Rm → R and f : [t0 , tf ] × Rn × Rm → Rn satisfy regularity conditions. Minimize Z tf J(u) = f0 (t, x(t), u(t)) dt t0

subject to x(t) ˙ = f (t, x(t), u(t)) and u(t) ∈ Ω for t ∈ [t0 , tf ], x(t0 ) = x0 , x(tf ) = xf , and u ∈ PC m [t0 , tf ]. Transformed Problem Minimize J(u) = x0 (tf ) subject to 

         x˙ 0 (t) f (t, x(t), u(t)) x (t ) 0 x0 (tf ) = 0 , 0 0 = , gf = 0, x(t) ˙ f (t, x(t), u(t)) x(t0 ) x0 x(tf )

    etc., where gf (x) = 0n×1 I x − xf with rank 0n×1 I = n.

Pontryagin’s Minimum Principle Fixed End Point Problems (Cont’d) EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Pontryagin’s Minimum Principle: Fixed End Point Suppose u ∗ ∈ U is optimal for the fixed end point problem. Then there exists a p0 ∈ {0, 1} such that the Hamiltonian H(t, x, u, p) = p0 f0 (t, x, u) + p T f (t, x, u) satisfies the minimum principle H(t, x ∗ (t), u ∗ (t), p ∗ (t)) = inf v ∈Ω H(t, x ∗ (t), v , p ∗ (t)) for all t ∈ [t0 , tf ] except possibly for a finite set, where x˙ ∗ (t) = f (t, x ∗ (t), u ∗ (t)), ∂f0 ∂f p˙ ∗ (t) = − (t, x ∗ (t), u ∗ (t))T p0 − (t, x ∗ (t), u ∗ (t))T p ∗ (t) ∂x ∂x for t ∈ [t0 , tf ], with x ∗ (t0 ) = x0 , x ∗ (tf ) = xf , and (p0 , p ∗ (tf )) ∂f 0 6= (0, 0). Moreover, if ∂f ∂t and ∂t are identically zero, then inf v ∈Ω H(·, x ∗ (·), v , p ∗ (·)) is constant on [t0 , tf ].

VII. Pontryagin’s Minimum Principle EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

1 Calculus of Variations

Fixed End Point Problems Examples

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

2 Pontryagin’s Minimum Principle

Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Pontryagin’s Minimum Principle Free Terminal Time Problems EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Free Terminal Time Problem Let f0 : [t0 , tf ] × Rn × Rm → R and f : [t0 , tf ] × Rn × Rm → Rn satisfy regularity conditions. Assume ∂f0 /∂t and ∂f /∂t exist and are continuous. Minimize Z tf

J(tf , u) =

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

f0 (t, x(t), u(t)) dt

t0

subject to x(t) ˙ = f (t, x(t), u(t)) and u(t) ∈ Ω for t ∈ [t0 , tf ], x(t0 ) = x0 , x(tf ) = xf , tf ∈ [t0 , ∞), and u ∈ PC m [t0 , tf ]. Transformed Problem With z0 (s) = x0 (t(s)), z(s) = x(t(s)), and v (s) = u(t(s)) for s ∈ [0, 1], minimize J(u) = z0 (1) subject to 

t˙ (s) z˙ 0 (s) z(s) ˙



=



α αf0 (t(s),z(s),v(s)) αf (t(s),z(s),v(s))

  t(0)  h i  t(1)  t0 z (0) z0 (1) = 0, , 0 = 0 , gf x0

z(0)



z(1)

 etc., and α ∈ (0, ∞), where gf (x) = 0n×2 I x − xf .

Pontryagin’s Minimum Principle Free Terminal Time Problems (Cont’d) EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Pontryagin’s Minimum Principle: Free Terminal Time Suppose (tf∗ , u ∗ ) ∈ [t0 , ∞) × U is optimal for the free terminal time problem. Then there exists a p0 ∈ {0, 1} such that H(t, x, u, p) = p0 f0 (t, x, u) + p T f (t, x, u) satisfies inf v ∈Ω H(tf∗ , x ∗ (tf∗ ), v , p ∗ (tf∗ )) = 0, H(t, x ∗ (t), u ∗ (t), p ∗ (t)) = inf v ∈Ω H(t, x ∗ (t), v , p ∗ (t)) for all t ∈ [t0 , tf∗ ] except possibly for a finite set, where x˙ ∗ (t) = f (t, x ∗ (t), u ∗ (t)), ∗ ∗ T 0 p˙ ∗ (t) = − ∂f ∂x (t, x (t), u (t)) p0 −

∗ ∗ T ∗ ∂f ∂x (t, x (t), u (t)) p (t) for t ∈ [t0 , tf∗ ], with x ∗ (t0 ) = x0 , x ∗ (tf∗ ) = xf , and (p0 , p ∗ (tf )) ∂f 0 6= (0, 0). Moreover, if ∂f ∂t and ∂t are identically zero, then inf v ∈Ω H(·, x ∗ (·), v , p ∗ (·)) is identically zero on [t0 , tf∗ ].

Pontryagin’s Minimum Principle Free Terminal Time Problems (Cont’d) EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Pontryagin’s Minimum Principle: Minimum Time Problem Suppose (tf∗ , u ∗ ) ∈ [t0 , ∞) × U is optimal for the free terminal time problem with f0 (t, x, u) = 1 for all (t, x, u). Let x ∗ solve x˙ ∗ (t) = f (t, x ∗ (t), u ∗ (t)),

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

t ∈ [t0 , tf∗ ],

with x ∗ (t0 ) = x0 and x ∗ (tf∗ ) = xf , and let p ∗ solve ∂f p˙ ∗ (t) = − ∂x (t, x ∗ (t), u ∗ (t))T p ∗ (t),

t ∈ [t0 , tf∗ ],

with p ∗ (tf ) 6= 0. Then inf p ∗ (tf∗ )T f (tf∗ , x ∗ (tf∗ ), v ) ∈ {−1, 0},

v ∈Ω

p ∗ (t)T f (t, x ∗ (t), u ∗ (t)) = inf v ∈Ω p ∗ (t)T f (t, x ∗ (t), v ) for all t ∈ [t0 , tf∗ ] except possibly for a finite set. Moreover, if ∂f ∗ T ∗ ∂t is identically zero, then inf v ∈Ω p (·) f (·, x (·), v ) is ∗ constant on [t0 , tf ].

VII. Pontryagin’s Minimum Principle EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

1 Calculus of Variations

Fixed End Point Problems Examples

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

2 Pontryagin’s Minimum Principle

Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Pontryagin’s Minimum Principle Examples EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Linear Quadratic Regulator:
f0 (t, x, u) = 12 x T L(t)x + u T R(t)u , ψ(x) = 12 x T Qx; f (t, x, u) = A(t)x + B(t)u, Q ≥ 0, L(t) ≥ 0, R(t) > 0; H(t, x, u, p) = f0 (t, x, u) + p T f (t, x, u), Ω = Rm Minimum principle: H(t, x ∗ (t), u ∗ (t), p ∗ (t)) = inf v ∈Ω H(t, x ∗ (t), v , p ∗ (t)) ⇒ u ∗ (t) = −R(t)−1 B(t)T p ∗ (t), t ∈ [t0 , tf ] \ (D ∗ ∪ D) Hamiltonian system: x˙ ∗ (t) = A(t)x ∗ (t) − B(t)R(t)−1 B(t)T p ∗ (t), x(t0 ) = x0 , ∗ T ∗ ∗ p˙ ∗ (t) =Qx  ∗= −L(t)x   (t) − A(t) p (t),−1p(tf ) T  ∗ (tf ) x˙ (t) A(t) −B(t)R(t) B(t) x (t) ⇒ ∗ = T p˙ (t) −L(t) −A(t) p ∗ (t) | {z } Hamiltonian matrix H(t)

Let Ψ(t, s) be the state transition matrix of H(·). Then

Pontryagin’s Minimum Principle Examples (Cont’d) EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Linear Quadratic Regulator (Cont’d):  ∗   ∗    x (t) x (tf ) I ∗ = Ψ(t, tf ) = Ψ(t, tf ) x (tf ) p ∗ (t) Qx ∗ (tf ) Q   X(t) ∗ = x (tf ) with X(t) invertible ∀t ≤ tf . P(t) Differential Riccati equation: p ∗ (t) = P(t)x ∗ (tf ) = P(t)X(t)−1 x ∗ (t) = K(t)x ∗ (t), ˙ K(t) = −A(t)T K(t) − K(t)A(t) +K(t)B(t)R(t)−1 B(t)K(t) − L(t), K(tf ) = Q. Candidate optimal control: u ∗ (t) = −R(t)−1 B(t)T p ∗ (t) = −R(t)−1 B(t)T K(t)x ∗ (t) for t ∈ [t0 , tf ] \ (D ∗ ∪ D). ⇒ If an optimal control exists, then u ∗ is optimal. (We have seen earlier that u ∗ is indeed optimal.)

Pontryagin’s Minimum Principle Examples (Cont’d) EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Min-Time Control of Rocket Car: f (t, x, u) = Ax +Bu,   0 1 0 where A = ,B= ; 0 0 1 t0 = 0, xf = 0, Ω = [−1, 1].

u(t)

0

>>

y(t)

Minimum principle: ∗ (t), u ∗ (t)) = inf ∗ T ∗ p ∗ (t)T f (t, x( v ∈Ω p (t) f (t, x (t), v ) ∗ −1 if p2 (t) > 0; ⇒ u ∗ (t) = t ∈ [t0 , tf∗ ] \ (D ∗ ∪ D). +1 if p2∗ (t) < 0; Adjoint equation: p˙ ∗ (t) = −AT p ∗ (t) ⇒ p2∗ (t) = p1∗ (tf∗ )(tf∗ − t) + p2∗ (tf∗ ). p2∗ (t) monotonic if p1∗ (tf∗ ) 6= 0; p2∗ (t) = p2∗ (tf∗ ) 6= 0 if p1∗ (tf∗ ) = 0. ⇒ |u ∗ (t)| = 1, ∀t ∈ [0, tf∗ ], and u ∗ changes sign at most once.

Pontryagin’s Minimum Principle Examples (Cont’d) EE 581 Optimal Control VII. Pontryagin’s Minimum Principle

Calculus of Variations Fixed End Point Problems Examples

Minimum Principle Minimum Terminal Cost Problems Free End Point Problems Fixed End Point Problems Free Terminal Time Problems Examples

Min-Time Control of Rocket Car (Cont’d): ∗ Case  ∗ I: u (t) = +1, ∀t.  

1  Rt 1 t −τ 0 x1 (t) (t − tf∗ )2 2 = t∗ dτ = x2∗ (t) f 0 1 1 t − tf∗ 1 ∗ 2 ∗ ∗ ∗ ⇒ x2 (t) = t − tf , x1 (t) = 2 x2 (t) .

Case II: u(t) = −1, ∀t. ⇒ x2∗ (t) = tf∗ − t, x1∗ (t) = − 12 x2∗ (t)2 .

Case III: u(t) = −1, ∀t < ˆt ; u(t) = 1, ∀t > ˆt . ⇒ x2∗ (t) − x2∗ (ˆt ) = ˆt − t, x1∗ (t) − x1∗ (ˆt ) = − 21 x2∗ (t)2 + 12 x2∗ (ˆt )2 .

Case IV: u(t) = +1, ∀t < ˆt ; u(t) = −1, ∀t > ˆt .

⇒ x2∗ (t) − x2∗ (ˆt ) = t − ˆt , x1∗ (t) − x1∗ (ˆt ) = 21 x2∗ (t)2 − 12 x2∗ (ˆt )2 .

Candidate  optimal control:

1 ∗ ∗ ∗ 2  −1 if x2 (t) ≥ 0 and x1 (t) ≥ − 2 x2 (t) ∗ 1 ∗ ∗ ∗ u (t) = or if x2 (t) < 0 and x1 (t) > 2 x2 (t)2 ;   1 otherwise.