ECE580 Fall 2015

Solution to Problem Set 2

October 5, 2015

1

ECE580 Solution to Problem Set 2 These problems are from the textbook by Chong and Zak, 4th edition, which is the textbook for the ECE580 Fall 2015 semester. As such, many of the problem statements are taken verbatim from the text; however, others have been reworded for reasons of efficiency or instruction. Solutions are mine. Any errors are mine and should be reported to me, [email protected], rather than to the textbook authors. 5.9 Level Sets The functions f1 : R2 → R and f2 : R2 → R are defined by f1 (x) = x21 − x22 , f2 (x) = 2x1 x2 .

(1) (2)

Draw the level sets C1 = {x : f1 (x) = 12} and C2 = {x : f2 (x) = 16} on the same set of axes. Identify on the diagram the set of x such that f (x) = [12, 16]T . Solution: The p solution is obtained using Matlab. The set C1 is the set of x such that x2 = ± x21 − 12. The set C2 is the set of x such that x2 = 8/x1 . For x1 6= 0 the intersection of these two level sets is the set of points x such that x21 −12−64/x21 = 0. Let y = x21 . Then the set of x such that x21 − 12 − 64/x21 = 0 corresponds to x41 − 12x21 − 64 = (x21 − 16)(x21 + 4) = 0. The imaginary roots are x1 = ±2j. The real roots are x1 = ±4. The corresponding real values of x2 are ±2. The intersecting level sets are shown in Figure 1. Figure 1 was generated by the following script: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%% %%%%% ECE569 Fall 2015, IUPUI %%%%% Plot of Intersecting Level Sets for Problem 5.9 of Chong %%%%% and Zak 4th ed. %%%%% c. 2015 S. Koskie %%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% clf clear x1a x1b x1c x2f1p1 x2f1p2 x2f1n1 x2f1n2 x2f2 x1a = -8:.01:-3.46; x2f1p1 = real(sqrt(x1a.^2-12)); x2f1n1 = -real(sqrt(x1a.^2-12));

ECE580 Fall 2015

Solution to Problem Set 2

October 5, 2015

2

Problem 5.9 Level Sets 8

6 f1(x1,x2) = x21 − x22 = 12 4

x2

2

0

−2

−4 f (x ,x ) = 2x x = 16 2

1 2

1 2

−6  2015 S. Koskie −8 −8

−6

−4

−2

0 x1

2

4

6

8

Figure 1: Intersecting Level Sets for Problem 5.9 x1b = 3.46:.01:8; x2f1n2 = -real(sqrt(x1b.^2-12)); x2f1p2 = real(sqrt(x1b.^2-12)); x1c = -8:.01:8; x2f2 = 16./(2*x1c); figure(5) plot(x1a,x2f1p1,’b’) hold plot(x1a,x2f1n1,’b’) plot(x1b,x2f1n2,’b’) plot(x1b,x2f1p2,’b’) plot(x1c,x2f2,’r’) axis([-8,8,-8,8]) xlabel(’x_1’) ylabel(’x_2’) title(’Problem 5.9 Intersecting Level Sets’) text(-5,5,’\color{blue}f_1(x_1,x_2) = x_1^2 - x_2^2 text(-1,-5,’\color{red}f_2(x_1,x_2) = 2x_1x_2 = 16’) text(4,-7.5,’\copyright 2015 S. Koskie’) grid print -depsc p5p9.eps hold

= 12’);

5.10 Taylor Series Expansions Find the Taylor series expansion of f (x) about x0 neglecting terms of order three and higher.

ECE580 Fall 2015

Solution to Problem Set 2

October 5, 2015

3

 T (a) f (x) = x1 e−x2 + x2 + 1 and x0 = 1 0 . Solution: We need the following factors that occur in the terms of the zeroth to second order terms of the Taylor series. f (x0 ) = 1 + 0 + 1 = 2. Df (x) =

h

∂f ∂x1

∂f ∂x2

i

=



e−x2 (−x1 e−x2 + 1)



so Df (x0 ) = " 2

F (x) = D f (x) =



1 (−1 + 1)

∂2f ∂x21 ∂2f ∂x1 ∂x2

∂2f ∂x1 ∂x2 ∂2f ∂x22



=



#

 =

1 0



.

0 −e−x2 −e−x2 x1 e−x2



so  F (x0 ) =

0 −1 −1 1

 .

Thus near x0 f (x) ≈ f (x0 ) + Df (x0 )(x − x0 ) + (x − x0 )T F (x0 )(x − x0 )/2 (3)    0 −1 x1 − x1,0 1 = 2 + x1 − x1,0 + [x1 − x1,0 x2 − x2,0 ] −1 1 x2 − x2,0 2 = 2 + x1 − 1 + (x1 − 1)(−x2 )/2 + x2 (−x1 + x2 + 1)/2 = 1 + x1 − x1 x2 + x2 + x22 /2. We check that the right hand side (rhs) equals f (x0 ) at x0 . The value of the rhs is 1 + 1 − 0 + 0 + 0 = 2 as expected.  T (b) f (x) = x41 + 2x21 x22 + x42 and x0 = 1 1 . Solution: We simplify this by noting that f (x) = (x21 + x22 )2 . Thus f (x0 ) = 22 = 4. Df (x) =



22 x1 (x21 + x22 ) 22 x2 (x21 + x22 )

so   Df (x0 ) = 8 8 .   4(3x21 + x22 ) 8x1 x2 F (x) = 8x1 x2 4(x21 + 3x22 )



ECE580 Fall 2015

Solution to Problem Set 2

so

 F (x0 ) =

16 8 8 16

October 5, 2015

4

 .

Thus near x0 f (x) ≈ f (x0 ) + Df (x0 )(x − x0 ) + (x − x0 )T F (x0 )(x − x0 )/2    16 8 x1 − 1 1 = 4 + 8(x1 − 1) + 8(x2 − 1) + [x1 − 1 x2 − 1] 8 16 x2 − 1 2 = −12 + 8x1 + 8x2 + (16(x1 − 1)2 + 16(x1 − 1)(x2 − 1) + 16(x2 − 1)2 )/2 = −12 + 8((x1 + x2 ) + (x21 − 2x1 + 1) + (x1 x2 − (x1 + x2 ) + 1) + (x22 − 2x2 + 1)) = 12 − 16(x1 + x2 ) + 8(x21 + x1 x2 + x22 ) We check that the right hand side (rhs) equals f (x0 ) at x0 . The value of the rhs is 12 − 32 + 24 = 4 as expected.  T (c) f (x) = ex1 −x2 + ex1 +x2 + x1 + x2 + 1 and x0 = 1 0 . Solution: We simplify this by noting that f (x) = ex1 (e−x2 + ex2 ) + x1 + x2 + 1. Thus f (x0 ) = e(2) + 1 + 0 + 1 = 2(e + 1).   Df (x) = ex1 (e−x2 + ex2 ) + 1 ex1 (−e−x2 + ex2 ) + 1 so Df (x0 ) =  F (x) =



 2e + 1 1 .

ex1 (e−x2 + ex2 ) ex1 (−e−x2 + ex2 ) ex1 (−e−x2 + ex2 ) ex1 (e−x2 + ex2 )

so

 F (x0 ) =

2e 0 0 2e



 .

Thus near x0 f (x) ≈ f (x0 ) + Df (x0 )(x − x0 ) + (x − x0 )T F (x0 )(x − x0 )/2      2e 0 x1 − 1 1 = 2(e + 1) + (2e + 1)(x1 − 1) + x2 + x1 − 1 x2 0 2e x2 2 2 2 = 2(e + 1) + (2e + 1)x1 + x2 − (2e + 1) + e(x1 − 1) + ex2 = 1 + (2e + 1)x1 + x2 + e(x21 − 2x1 + 1 + x22 ) = 1 + e + x1 + x2 + ex21 + ex22 We check that the right hand side (rhs) equals f (x0 ) at x0 . The value of the rhs is 1 + e + 1 + 0 + e + 0 = 2(e + 1) as expected.

ECE580 Fall 2015

Solution to Problem Set 2

October 5, 2015

5

6.1 Consider the problem minimize f (x) subject to x ∈ ω, where f ∈ C 2 . For each of the following definitions of Ω, x∗ , and f , determine whether x∗ is, is not, or might be a local minimizer.  T  T (a) f : R2 → R, Ω = {x : x1 ≥ 1}, x∗ = 1 2 , ∇f (x∗ ) = 1 1 . ∗ T ∗ Solution: x is on the boundary so we must verify that d ∇f (x ) ≥ 0 for all feasible directions d. To be feasible, d1 must be nonnegative, and d2 is arbitrary, so long as d 6= 0. dT ∇f (x∗ ) = d1 + d2 . Since d2 is arbitrary, this quantity may be negative, thus x∗ is not a local minimizer.  T  T (b) f : R2 → R, Ω = {x : x1 ≥ 1, x2 ≥ 2}, x∗ = 1 2 , ∇f (x∗ ) = 1 0 . ∗ Solution: In this case, x is on the boundary at a corner. Thus, to be feasible, d must have both d1 and d2 nonnegative. dT ∇f (x∗ ) = d1 ≥ 0, so the FONC is satisfied and x∗ could be a local minimizer.  T  T (c) f : R2 → R, Ω = {x : x1 ≥ 0, x2 ≥ 0}, x∗ = 1 2 , ∇f (x∗ ) = 0 0 , ∗ F (x ) = I. Solution: x∗ is an interior point. Since the gradient is zero at x∗ , the FONC is satisfied. Since the Hessian is positive definite, the SOSC is satisfied so x∗ is a local minimizer.  T  T (d) f : R2 → R, Ω = {x : x1 ≥ 1, x2 ≥ 2}, x∗ = 1 2 , ∇f (x∗ ) = 1 0 ,  T 1 0 F (x∗ ) = . 0 −1 Solution: x∗ is on the boundary at a corner. Both d1 and d2 must be nonnegative as before, and not both zero. The Hessian is not positive definite, and dT F (x∗ )d = d21 − d22 . The SONC is not satisfied for any d having d2 > d1 . Thus, regardless of whether the FONC is satisfied, x∗ is not a local minimizer. 6.8 Gradients, Directional Derivatives, and Hessians, FONC and SONC Consider the following function f : R2 → R: f (x) = x

T



1 2 4 7

 x+x

T



3 5

 + 6.

ECE580 Fall 2015

Solution to Problem Set 2

October 5, 2015

6

To simplify the calculations, we symmetrize to obtain     1 3 1 2 T T x, x=x x 3 7 4 7 and rewrite the function as f (x) = x

T



1 3 3 7

 x+x

T



3 5

 + 6.

 T . (a) Find the gradient and Hessian of f at x0 = 1 1 Solution: The gradient is     1 3 3 ∇f (x) = 2 x+ , 3 7 5 which, evaluated at x0 is  ∇f (x0 ) = 2

1 3 3 7



The Hessian is 2

D f =2

1 1 



 +

1 3 3 7

3 5



 =

11 25

 .

 ,

which is a constant, so independent of x0 . (b) Find the directional derivative of f at x0 with respect to a unit vector in the direction of maximal rate of increase. Solution: The direction of maximum increase of f at a point is the value of the gradient at that point, in this case ∇f (x0 ). The directional derivative with along the direction d is ∂f (x) = dT ∇f (x) ∂d . Normalizing, we obtain     p 1 1 11 11 T ˆ d = ∇f (x0 )/ (∇f (x0 )) ∇f (x0 ) = √ =√ . 121 + 625 25 746 25 Thus the directional derivative with respect to a unit vector along the maximal rate of increase is p (∇f (x))T ∇f (x) p = (∇f (x))T ∇f (x). (4) (∇f (x))T ∇f (x) At x0 we have .

√ ∂f (x0 ) = 746 ∂ dˆ

ECE580 Fall 2015

Solution to Problem Set 2

October 5, 2015

7

(c) Find a point that satisfies the FONC and determine whether it satisfies the SONC. Solution: Such a point must satisfy ∇f (x) = 0, so we need   ∗    1 3 x1 3 ∇f (x0 ) = 2 + = 0. ∗ 3 7 x2 5 One such x∗ =



3/2 −1

T

. 

To determine whether it satisfies the SONC we must determine whether

2 6 6 14

is positive definite. Solving for the eigenvalues we obtain   s−2 −6 2 −6 s − 14 = (s − 2)(s − 14) − 36 = s − 16s − 12. Using the quadratic formula we find that √ √ s = (16 ± 176 + 48)/2 = 8 ± 4 14, so the matrix is indefinite and the point does not satisfy the SONC. A more efficient way to determine this for a two by two matrix is to take the determinant. The determinant is the product of the eigenvalues, so being negative indicates that the two eigenvalues have opposite signs. 6.10 Finding Minimizers Consider the function f : R2 → R    2 5 3 T f (x) = x x+x +7 −1 1 4     2 2 3 T T = x x+x +7 2 1 4 T



 (a) Find the directional derivative of f at x0 = Solution

 ∇f (x) = 2

2 2 2 1

0 1



 in the direction d = 

x+



3 4

 .

The value of the requested directional derivative is thus         4 4 0 3 T 1 0 d ∇f (x0 ) = + 4 2 1 4 = 7.

1 0

 .



ECE580 Fall 2015

Solution to Problem Set 2

October 5, 2015

8

(b) Find all points that satisfy the FONC. Solution Points that satisfy the FONC are points at which the gradient is zero.     4 4 3 ∗ ∇f (x) = x + 4 2 4 has solution x

∗



−1 

 −3 = −4    −1/4 1/2 −3 = 1/2 −1/2 −4   −5/4 = . 1/2 4 4 4 2

Because the inverse exists, the result is unique. The SONC is not satisfied because the determinant of the Hessian is negative, so the two by two matrix has one positive and one negative eigenvalue, hence is indefinite. Accordingly, f does not have a minimizer. 6.16 Constrained Feasible Set Consider the problem minimize f (x) subject to x ∈ Ω where f (x) = 4x21 − x22 , and Ω = {x : x21 + 2x1 − x2 ≥ 0, x1 ≥ 0, x2 ≥ 0}. (a) Does the point x∗ = 0 satisfy the FONC? Solution The origin is on the boundary of Ω, in fact on a corner. Thus we must test all feasible directions to determine whether dT ∇f (x∗ ) ≥ 0 for all of them. The slope of the curve x2 = x21 + 2x1 is 2x1 + 2. This yields a slope of 2  T at the origin, so we must consider all directions d1 d2 for which d2 ≥ 2d1 down to the horizontal. The condition for the FONC is thus   8x1 T d ≥ 0. 4x2 In fact, since we are looking at x∗ , the gradient is zero so the FONC is satisfied for all feasible directions. (b) Does the point x∗ = 0 satisfy the SONC?   8 0 Solution: The Hessian is , independent of the value of mathbf x∗ . 0 −2   8 0 T d d = 8d21 − 2d22 . 0 −2

ECE580 Fall 2015

Solution to Problem Set 2

October 5, 2015

T

Now so long as d2 ≤ 2d1 , we will have d



8 0 0 −2

9

 d ≥ 0 so the SONC is

satisfied. (c) Is the point x∗ = 0 a local minimizer of f over Ω? A local minimizer of f over Ω, could occur anywhere on the boundary or the interior of Ω. Because the Hessian is indefinite everywhere, the minimizer does not occur in the interior of Ω. If we check the section of the boundary along the x1 axis, the FONC is satisfied for any direction vector that has d2 = 0. However, this does not cover all feasible directions, so the FONC is not satisfied for nonzero points on the positive x1 axis. The last section of the boundary is the curve x2 = x21 + 2x1 . (x21 + 2x1 )2 = x41 + 4x31 + 4x21 , so on this boundary, f (x) = 4x21 − 2(x21 + 2x1 )2 = −2x41 − 8x31 , which is negative for all positive x1 as one approaches the origin along the curve. Thus the origin is not a local minimizer of f over Ω. 7.8 Application of Newton’s Method of Tangents to Root Finding x

−1 Let g(x) = eex +1 , x ∈ R. Notice that the denominator is never zero and that the unique zero of the numerator, hence g is x = 0.

(a) Apply Newton’s Method of Tangents Solution: The algorithm is x(k+1) = x(k) −

g(x(k) ) g 0 (x(k) )

(5)

where g 0 := dg/dx. d x dg (x) = (e − 1)(ex + 1)−1 dx dx = ex (ex + 1)−1 + (ex − 1)(−1)(ex )(ex + 1)−2 ex (ex + 1) − ex (ex − 1) = (ex + 1)2 2ex = . (ex + 1)2 We will be dividing by this quantity, so we must verify that it is never zero. The numerator is indeed never zero. Substituting into (5), we obtain ! ! x(k) x(k) 2 e − 1 (e + 1) x(k+1) = x(k) − ex(k) +1 2ex(k) (k)

1 − e2x = x + 2ex(k) (k) (k) e−x − ex (k) = x + 2 = x(k) − sinh x(k) . (k)

ECE580 Fall 2015

Solution to Problem Set 2

October 5, 2015

10

(b) Find an initial condition x(0) such that the algorithm cycles, specifically such that x(0) = x(2) = x(4) = · · · . Solution The graph of sinh x is odd, i.e. sinh(−x) = − sinh(x), see Figure 2. Thus for the algorithm to cycle such that x(2k) = x(0) for all nonnegative integers k, we need x(k+1) = −x(k) . This is true when x(k+1) = −x(k) = x(k) − sinh x(k) . Equivalently, we need 2x(k) = sinh xk , and, in particular 2x(0) = sinh x(0) (0)

, which, according to Matlab occurs when x(0) = x∗ ≈ ±2.1773. 4

1.5

Hyperbolic Sine Function

x 10

1

sinh(x)

0.5

0

−0.5

−1

−1.5 −10

−8

−6

−4

−2

0 x

2

4

6

8

10

Figure 2: Graph of the odd function sinh(x) (0)

(c) If kx0 k > x∗ , then kx(k+1) k > kx(k) k for all nonnegative integers k and the sequence generated by the algorithm diverges. Examples of convergence, cycling, and divergence are shown in Figures 3 through 5. This is a good example showing why a picture is not a proof. The value of x(0) used for the cycling example is an approximation, chosen carefully so that the algorithm would appear to cycle for the ten iterations shown. It converges or diverges later. It is (0) likely that the actual value of x∗ is irrational so can never be captured with finite precision. 7.10

ECE580 Fall 2015

Solution to Problem Set 2

October 5, 2015

(0)

Newton Algorithm Converging for x =2 2

1.5

1

x(k)

0.5

0

−0.5

−1

−1.5

−2

1

2

3

4

5

6 k

7

8

9

10

11

Figure 3: ’Fast convergence of Newton Algorithm for Good x(0) ’

(0)

Newton Algorithm Cycling for x

= 2.177319

2.5

2

1.5

1

x(k)

0.5

0

−0.5

−1

−1.5

−2

−2.5

1

2

3

4

5

6 k

7

8

9

10

11

Figure 4: Apparent Cycling of Newton Algorithm’

11

ECE580 Fall 2015

Solution to Problem Set 2

(0)

Newton Algorithm Diverging for x

October 5, 2015

= 2.177348

250

200

x(k)

150

100

50

0

−50

1

2

3

4

5

6 k

7

8

9

10

11

Figure 5: Divergence of Newton Algorithm for Bad x(∗) ’

12