Lecture 2. Solving Linear Systems As we discussed before, we can solve any system of linear equations by the method of elimination, which is equivalent to applying a sequence of elementary row reductions over its augmented matrix. Some terminologies: Leading entry –the …rst nonzero entry of a row (or column) is called the leading entry of the row (or column).

Echelon Form–A matrix is called in Echelon form (upper triangle form) if (a) All non-zero rows are above any zero-row (row with all entries zero) (b) For any two rows, the column containing the leading entry of the upper row is on the left of the column containing the leading entry of the lower row. In short, Echelon form displays upper-triangle pattern. In an Echelon form, leading entries of rows form a stair-shape. For instance (where * indicates a leading entry of a row), 2

1 Echelon form: 4 0 0

2 0 0

1 8 0 2

3 2 1 3 0 1 8 5; 4 0 0 0 0

0 non-Echelon form: 4 2 0

2 0 0

2 4 0

1 3 1 1 0 4

3 1 3 0 8 1 8 5: 0 0 0

3 0 8 5 0

Reduced Echelon Form –a matrix is called Reduced Echelon form If, in addition to (a) and (b) above, (c) all leading entries are 1 and it is the only non-zero entry in each pivot column. For instance, the followings are Reduced 2 3 2 1 0 0 3 0 1 4 0 0 1 1 8 5; 4 0 0 0 0 0 0 0

Echelon forms: 3 0 1 0 0 1 1 0 8 5: 0 0 1 0

Pivot –leading entry 1 in the reduced echelon form is called pivot position (or pivot entry.)

Pivot Column – column containing a pivot position is called pivot column.

1

Theorem 2.1. Any matrix can be reduced by a sequence of elementary row operations to a unique reduced Echelon form. Theorem 2.2. Elementary row operations do not a¤ect the solution set of any linear system. Consequently, the solution set of a system is the same as that of the system whose augmented matrix is in the reduced Echelon form. The system can be solved from bottom up once it is reduced to an Echelon form.

Gauss-Jordan Algorithm Gauss-Jordan Algorithm is a classic algorithm of …nding reduced Echelon form. It consists of several steps described as follows. Step 1. From the left, …nd the …rst non-zero column (it is a pivot column). By interchanging two rows if necessary, make sure that the leading entry of the column is non-zero. In other words, by performing row operations, we end up with the following matrix (* represents any number) 3 3 2 2 0 ::: 0 a ::: a ::: 6 0 ::: 0 b 6 b ::: 7 ::: 7 7 7 6 6 4 ::: ::: ::: ::: 5 or 4 ::: ::: ::: ::: ::: ::: 5 ; a 6= 0: 0 ::: 0 c ::: c ::: Step 2. Perform elementary row operation #3: R1 =a ! R1 ; we arrive at 2 3 1 =a =a ::: 6 b ::: 7 7: 6 4 ::: ::: ::: ::: 5 c :::

Then, perform elementary row operation #1 several times (i.e., …rst, R2 bR1 ! R2 ; ..., …nally Rm cR1 ! Rm ), we obtain a matrix 2 3 1 ::: 6 0 A B ::: 7 7 6 4 ::: ::: ::: ::: 5 : 0 C :::

Step 3. Repeat the above two steps for the submatrix obtained from deleting the …rst row, 2 3 0 A B ::: 4 ::: ::: ::: ::: 5 : 0 C ::: This sub-matrix is then reduced 2 0 6 0 6 4 ::: 0

row-equivalently to 3 1 E ::: 0 ::: 7 7; ::: ::: ::: ::: 5 0 G ::: 2

and the original matrix is reduced 2 1 6 0 6 4 ::: 0

to 3 ::: E ::: 7 7: ::: ::: 5 G :::

W 1 ::: 0

By perform row operation: R1 W R2 ! R1 ; 2 1 0 0 6 0 1 6 4 ::: ::: ::: ::: 0 0

we have 3 ::: ::: 7 7: ::: 5 :::

Step 4. Repeat till we are forced to stop – this …nal matrix is the reduced Echelon form. Example 2.1. Find the reduced Echelon form for 3 2 1 2 1 j 0 4 0 2 8 j 8 5 4 5 9 j 9

and then solve the corresponding linear system 8 x1 2x2 + x3 = 0 < 2x2 8x3 = 8 : : 4x1 + 5x2 + 9x3 = 9

Solution: We perform row operations on the augmented matrix as follows:

2

1 4 0 4

2 2 5

2

3 2 3 0 1 2 1 j 0 8 5 R3 + 4R1 ! R3 4 0 2 8 j 8 5 ! 9 0 3 13 j 9 3 2 2 1 j 0 1 0 7 j 1 4 j 4 5 R1 + 2R2 ! R1 4 0 1 4 j ! 3 13 j 9 0 3 13 j

1 j 8 j 9 j

1 R2 =2 ! R2 4 0 ! 0

2

1 0 R3 + 3R2 ! R3 4 0 1 ! 0 0 2 1 0 R2 + 4R3 ! R2 4 0 1 ! 0 0

3 2 7 j 8 1 0 4 j 4 5 R1 + 7R3 ! R1 4 0 1 ! 1 j 3 0 0 3 0 j 29 0 j 16 5 : 1 j 3

We obtain the reduced Echelon form 2 3 1 0 0 j 29 4 0 1 0 j 16 5 : 0 0 1 j 3 3

3 8 4 5 9

3 0 j 29 4 j 4 5 1 j 3

The linear system with the above matrix as its augmented matrix is 8 = 29 < x1 x2 = 16 : : x3 = 3 Solution: x1 = 29; x2 = 16; x3 = 3; or (29; 16; 3) : Solution Sets of Linear Systems As we discussed above, one can solve any system by reducing its augmented matrix to the reduced Echelon form. In fact, it is not necessary to get the reduced echelon form - any Echelon form is enough to solve the system. For instance, we can solve the same system in Example 2.1 as follows. First, we perform the same row operations to get an Echelon form 2 3 2 3 1 2 1 j 0 1 2 1 j 0 4 0 2 8 j 8 5 R3 + 4R1 ! R3 4 0 2 8 j 8 5 ! 4 5 9 j 9 0 3 13 j 9 2 3 2 3 1 2 1 j 0 1 2 1 j 0 4 j 4 5 R3 + 3R2 ! R3 4 0 1 4 j 4 5: R2 =2 ! R2 4 0 1 ! ! 0 3 13 j 9 0 0 1 j 3 Next, we write down the corresponding system 8 < x1 2x2 + x3 = 0 x2 4x3 = 4 : : x3 = 3

Finally, we solve the system bottom up: From the last equation, we see x3 = 3: Next, we substitute this into the second equation to get x2

4 3 = 4 =) x2 = 16:

We then substitute x3 = 3 and x2 = 16 into the …rst equation, we obtain x1

2 16 + 3 = 0 =) x1 = 29:

In summary, to solve a system of linear equations whose augmented matrix is M: We follow the following steps. Step 1: Perform elementary row operations to reduced M to its reduced Echelon form E. Step 2: In the matrix E; locate pivot positions and pivot columns. The variable associated with pivot columns are called basic variables. For instance, for 2 3 1 0 1 j 1 E = 4 0 1 5 j 6 5 ; x1 and x2 are basic variables: 0 0 0 j 0 4

Step 3: Write down the system of linear equations whose augmented matrix is E: Solve all pivot variables in terms of non-pivot variables. Those non-basic variables are referred as to free variables or parameters since they can be chosen freely to generate solutions. It is possible to solve a linear system using any Echelon form instead of its reduced Echelon form. To do so, the very last step needs to be modi…ed to Step 3 : Write down the system of linear equations whose augmented matrix is E: Solve all pivot variables in terms of non-pivot variables by backward substitutions. Example 2.2. Consider a linear system whose augmented matrix is reduced to an Echelon form 2 3 1 3 1 j 1 4 0 1 2 j 2 5: 0 0 0 j 0 Find the describe the solution set. Solution: Notice that the matrix has two pivot columns that are associated with variables x1 and x2 : These two variable are basic variables. x3 is the free variable. We write down the corresponding system 1

+ 3x2 x3 = 1 x2 + 2x3 = 2;

and solve, from bottom up, basic variables in terms of the free variable. In this case, from the second equation, we solve x2 in terms of x3 : x2 = 2

2x3 :

Next, we substitute it into the …rst equation x1 + 3 (2

2x3 )

x3 = 1

and solve for x1 as x1 = 13

(2

2x3 ) + x3 = 11 + 3x3 :

Here, x3 can be chosen arbitrarily: for any x3 = t; we have a solution x1 = 11 + 3t x2 = 2 2t x3 = t: Therefore, there are in…nite many solitons. For instance, take t = 0; x1 = 11; x2 = 2; x3 = 0; take t = 1 x1 = 14; x2 = 0; x3 = 1: 5

(1)

We call (1) a parametric representation of the solution set, or general solution. In general, we choose non-pivot variables as parameters and solve pivot variables in terms of non-pivot variables. Example 2.3 Solve x1

2x2 + x3 + x4 = 0 2x2 + 6x4 = 4

and describe general solution by a parametric representation. Solution: Its augmented matrix 1 0

2 2

1 1 j 0 0 6 j 4

is already in Echelon form, and x1 and x2 are pivot variables. So we …rst solve x2 from the second equation as x2 = 2

3x4 :

Then substitute it into the …rst equation and solve for x1 as x1 x1 = 2 (2

2 (2 3x4 )

3x4 ) + x3 + x4 = 0 x3

x4 = 4

x3

7x4 :

Set x3 = s; x4 = t; we …nd a parametric representation x1 x2 x3 x4

=4 =2 =s = t:

s 7t 3t

Consistency Not every system has a solution. For instance, consider a system of three equation with three unknowns. In 3D space, each linear equation represents a plane. So the system represents the intersection of three planes, which could be a single point, or a straight line, or a plane (all three planes are identical), or empty set (no solution). Theorem 2.3 (Consistency) A system is called consistent if it admits at least one solution. A system is consistent if and only if the rightmost column contains NO pivot. Proof: If the rightmost column is a pivot column, then in its reduced Echelon form, there is a pivot in the rightmost column: 2 3 1 ::: j 6 0 7 ::: j 6 7 4 0 0 ::: A j b 5 ; A = 0: 0 0 0 0 j 0 6

In other words, there is a row (the last not-all-zero row) having the form [0; 0; :::; 0; b], where b 6= 0: Accordingly, the very last non-trivial equation is 0 = b; but b 6= 0; A contradiction. Example 2.4. In the second example of 2 3 2 0 1 4 j 8 1 4 2 3 2 j 1 5!4 0 5 8 7 j 1 0

the previous section, we have 3 3=2 1 j 1=2 1 4 j 8 5: 0 0 j 5=2

The last column is a pivot column. Thus, it is inconsistent. Example 2.5. Determine if the system

3x1 3x1

3x2 6x3 + 6x4 + 4x5 = 5 7x2 + 8x3 5x4 + 8x5 = 9 9x2 + 12x3 9x4 + 6x5 = 15

Solution. The augmented matrix is 2 0 3 6 6 4 j 4 3 7 8 5 8 j 3 9 12 9 6 j

We perform following row operations: 2 0 3 6 6 4 j 4 3 7 8 5 8 j 3 9 12 9 6 j 2 3 7 8 6 R2 ! R1 4 0 3 ! 3 9 12 2 3 7 R 3 R1 ! R 3 4 0 3 ! 0 2 2 3 7 2 R3 + R2 ! R3 4 0 3 3 ! 0 0

3 5 9 5 15

5 6 9

8 6 4 8 6 0

8 j 4 j 6 j 5 6 4

3 5 9 5: 15

3 9 5 5 15

8 j 4 j 2 j

5 8 j 6 4 j 0 2=3 j

3 9 5 5 6

3 9 5 5:

We conclude that the linear system is consistent. When a linear system is consistence, from the above examples, we see that the system admits a unique solution if and only if there is no free variable. Theorem 2.4. Consider a linear system whose augmented matrix is M: There are three possibilities. (1) If the last column of M is a pivot column, then the system is inconsistent. (2) If the last column of M is a not pivot column but the rest columns are all pivot, then the system admits a unique solution. 7

(3) If, additional to the last column that is not pivot, there is at least one more non-pivot column, then the system has in…nite many solutions. Note that Gauss-Jordan Algorithm is the most basic algorithm. In Matlab or Mathematica, there are several build-in functions for solving linear systems. There are many more e¢ cient algorithm for solving linear systems. Here we introduce another popular algorithm. LU Decomposition Another classical method to solve linear system A~x = ~b is so-called LU decomposition: Find a lower triangle matrix L and a upper triangle matrix U such that 2 32 3 1 0 0 a11 6 1 6 7 07 7 6 0 a22 7 6 Am n = Lm m Um n = 6 . . . 7 6 7: . . . . .. .. . . .. 5 4 .. 4 .. .. . .. 5 0 0 1 Now the linear system A~x = ~b can be solved by solving two easier systems L~y = ~b U~x = ~y : The …rst system L~y = ~b can be solved from top to bottom, and the second system can be solved from bottom up. LU Decomposition Algorithm 1. Reduce A to an echelon form from U by a sequence of type one row operations (row replacement row operation) 2. Place entries in L such that the same sequence of row operations reduces L to the identity matrix. The rigorous justi…cation of this algorithm will be presented later in Lecture 7. Example 2. Find LU decomposition and 2 2 4 1 5 6 4 5 3 8 A=6 42 5 4 1 6 0 7 3

then solve for A~x = ~b : 3 2 3 2 1 607 17 7 ; ~b = 6 7 : 425 85 1 14

Solution: Obviously, since A is 4 5; L should be 4 4: The …rst step to reduce A to an echelon form is to replace the …rst column as follows 2 3 2 3 2 4 1 5 2 2 4 1 5 2 2R1 + R2 ! R2 6 6 4 5 3 8 17 1 2 37 7 R1 + R3 ! R3 60 3 7 = A1 : A=6 42 40 5 4 1 85 9 3 4 10 5 3R1 + R4 ! R4 6 0 7 3 1 4 12 5 ! 0 12 8

Therefore, 2

3 2 3 1 0 0 0 1 0 0 0 2R + R ! R 1 2 2 6 2 1 0 07 60 1 0 07 7 6 7 L=6 4 1 a 1 05 R1 + R3 ! R3 40 a 1 05 = L1 : 3R1 + R4 ! R4 3 b c 1 ! 0 b c 1

We repeat 2 2 60 A1 = 6 40 0

the process to …nd a; b : 3 2 4 1 5 2 2 4 6 3 1 2 37 3R + R ! R 0 2 3 3 7 6 3 9 3 4 10 5 4R2 + R4 ! R4 40 0 ! 0 0 12 4 12 5

and accordingly a = 3; b = 4 because 3 2 2 1 0 0 0 1 60 1 0 07 3R2 + R3 ! R3 60 7 6 L1 = 6 40 3 1 05 4R2 + R4 ! R4 40 ! 0 0 4 c 1 Finally, we …nd 2 2 60 A2 = 6 40 0

1 1 0 0

5 2 2 4

so c = 2 since

2 1 60 6 L2 = 4 0 0

0 1 0 0

0 0 1 2

We conclude

2

1 6 2 L=6 41 3

0 1 3 4

Now we solve

2

2 3 2 2 60 37 7 2R3 + R4 ! R4 6 15 ! 40 0 7

4 3 0 0

2 3 1 0 6 07 7 2R3 + R4 ! R4 60 05 ! 40 0 1

0 1 0 0

3 2 0 0 2 60 0 07 7; U = 6 40 1 05 2 1 0

1 6 2 L~y = 6 41 3

4 3 0 0

1 1 0 0

5 2 2 0

32 3 2 3 0 0 0 y1 1 6y2 7 6 0 7 1 0 07 76 7 = 6 7 3 1 05 4y3 5 4 2 5 4 2 1 y4 14 9

5 2 2 4

3 2 37 7 = A2 15 7

3 0 0 0 07 7 = L2 : 1 05 c 1

0 1 0 0

c: 4 3 0 0

1 1 0 0

1 1 0 0

0 0 1 0

5 2 2 0

3 0 07 7: 05 1 3 2 37 7: 15 5

3 2 37 7 15 5

from top down as y1 2y1 + y2 y1 3y2 + y3 3y1 + 4y2 + 2y3 + y4

=1 = 0 =) y2 = 2y1 = 2 = 2 =) y3 = 2 y1 + 3y2 = 2 1 + 6 = 7 = 14 =) y4 = 14 + 3y1 4y2 2y3 = 14 + 3

8

14 =

Finally, we solve 2 2 60 6 40 0

4 3 0 0

1 1 0 0

5 2 2 0

32 3 2 3 2 y1 2 x1 6x2 7 6y2 7 6 37 76 7 = 6 7 = 6 1 5 4x3 5 4y3 5 4 y4 x4 5

from bottom up ( x3 is free variable): x5 =

1

2x4 + x5 = 7 =) x4 = 3x2 + x3 + 2x4

2x1 + 4x2

x3 + 5x4

3 1 27 7 75 5

7

x5 2

2x4 + 3x5 3 2 x3 8 3 x3 = = 3 3 3 1 4x2 + x3 5x4 + 2x5 2x5 = 1 =) x1 = 2 x3 1 4 3 + x3 20 2 7 3 = = x3 2 6 3x5 = 2 =) x2 =

2

=4

x3

9 : 2

Remark: When there is a need to exchange two rows, the LU algorithm need to modi…ed accordingly. Homework #2 1. Determine whether each of the following matrices is in reduced Echelon form or only in Echelon form, or neither. If in Echelon form, indicate all pivot positions and pivot columns. 2 3 0 1 0 6 0 5 5 0 9 5 (a) 4 0 0 1 0 0 0 0 1 15 2 3 1 1 2 6 0 5 5 1 9 5 (b) 4 0 0 1 0 0 0 0 0 1 3 2 1 1 2 6 0 5 1 5 (c) 4 0 0 1 0 1 0 0 0 10

5:

2. Determine the value(s) of h such that the matrix is the augmented matrix of a consistent linear system. Then …nd the general solution. 2 3 1 1 2 6 0 4 1 0 1 5 h 5: 3 1 0 16 2 3. Find general solutions for linear systems whose augmented matrices are given: (a)

(b)

2

1 4 0 1

7 0 7

2

1 4 0 0

2 0 0

0 1 4 3 1 0

3 A=4 3 9

3 3 2 5: 3

1 0 1

1 2 2

1 2 6

3 2 105 6

4. Find LU Decomposition for

2

3 5 3 5: 7

6 2 2

make sure verify your answer by direct multiplication of LU: 5. For each of the following statements, determine whether it is true or false. If your answer is true, state your rationale. If false, provide an counterexample (the example contradicting the statement). (a) A matrix may be row reduced to more than one matrix in reduced echelon form, using di¤erent sequences of row operations. (b) A basic variable is a variable corresponding to a pivot columns (c) If one row in an echelon form of an augmented matrix is [0 0 0 0 0 3 0] ; then the associated system is inconsistent. (d) The pivot positions depend on how you do row reduction. (e) Whenever a system have more than one solution, it must has a free variable.

11