Review of Vector Spaces and Matrix Algebra Review of Linear Algebra Concepts concerning Vector Spaces is covered in Calculus III and therefore is a prerequisite for this class. We give a brief review here. Definition 1. A vector space V is a collection of objects, referred to as vectors, together with an operation of vector addition (which allows us to add two vectors together) and a scalar multiplication (which allows us to multiply a scalar times a vector). Here are two examples: Example 1. For example, V = Rn = {x = (x1 , x2 , · · · , xn ) : xj ∈ R for j = 1, · · · , n}. Here we define the vector addition and scalar multiplication as follows. 1. Given vectors x = (x1 , x2 , · · · , xn ) and y = (y1 , y2 , · · · , yn ) we define x + y by x + y = (x1 + y1 , x2 + y2 , · · · , xn + yn ). 2. Given a scalar α and a vector x = (x1 , x2 , · · · , xn ) we define αx by αx = (αx1 , αx2 , · · · , αxn ). Example 2. The set of continuous functions on an interval [a, b] = {x ∈ R : a ≤ x ≤ b} which we denote by C[a, b]. We have learned in calculus that the sum of two continuos functions is a continuous function and a constant times a continuous function is a continuous function. In order for a collection of vectors V with an addition and scalar multiplication to be a vector space the following Axioms must be satisfied: Vector Addition For every x = (x1 , x2 , · · · , xn ), y = (y1 , y2 , · · · , yn ), z = (z1 , z2 , · · · , zn ) ∈ V 1. (closure property) x + y ∈ V . 2. (commutative property) (x + y) = (y + x). 3. (associative law) (x + y) + z = x + (y + z) 4. (zero vector) There is a unique zero vector 0 ∈ V satisfying x + 0 = 0 + x = x. 5. (additive inverse) For every x ∈ V there exists a vector −x ∈ V satisfying x + (−x) = (−x) + x = 0. Scalar Multiplication For every x = (x1 , x2 , · · · , xn ), y = (y1 , y2 , · · · , yn ) ∈ V and scalars k, k1 and k2 1. (closure property) kx ∈ V . 2. (distributive law 1) k(x + y) = (ky + kx). 3. (distributive law 2) (k1 + k2 )x = k1 x + k2 x. 4. (distributive law 3) k1 (k2 x) = (k1 k2 )x. 5. (multiplicative identity) 1x = x. Definition 2. 1. A subset W of a vector space V (denoted W ⊂ V ) is called a subspace if it is closed under vector addition and scalar multiplication. 2. A collection of vectors {xj }nj=1 in a vector space V is said to be Linearly Independent if the only constants {kj }nj=1 satisfying k1 x1 + k2 x2 + · · · + kn xn = 0 are k1 = k2 = · · · = kn = 0. If a set of vectors is not linearly independent then we say it is Linearly Dependent.

1

3. A collection of linearly independent vectors {xj }nj=1 in a vector space V is said to be Basis for the vector space if every x ∈ V can be written as a linear combination of the vectors n X n n {xj }j=1 , i.e., given x ∈ V there are constants {cj }j=1 so that x = cj x j . j=1

4. The number of vectors in a basis is called the Dimension of the vector space. Example 3. A basis (called the standard basis) for Rn is       1 0 0 0 1 0             e1 = 0 , e2 = 0 , · · · , en =  ...  .  ..   ..    . . 0 0 0 1 Therefore the dimension of Rn is n. Example 4. The vector space C[a, b] is infinite dimensional. Remark 1. Given a collection of linearly independent vectors {xj }`j=1 in a vector space V we define the Span, S, of {xj }`j=1 to be the collection of all linear combinations of the vectors, i.e., S=

( ` X

) cj xj : for all scalars cj , j = 1, · · · , n .

j=1

The span of a set of vectors is a subspace and we can say that a basis is a linearly independent spanning set. Another important property of the vector space Rn is that we can do geometry by introducing the n X n so-called dot-product (or inner product). For x, y ∈ R we define x · y = xj yj . We will also j=1

often use the notation hx, yi for the inner product. The inner product allows us to consider tow very important things: the length of a vector and the angle between two vectors. In particular we define the length of a vector by 1/2

kxk = hx, xi

=

n X

!1/2 x2j

.

j=1

Notice this is exactly the formula given by the distance formula in calculus. Then given two vectors x and y we can consider the angle θ between the vectors and using the law of cosines we obtain a formula for cos(θ) as cos(θ) =

hx, yi . kxkkyk

2

Matrices Section 8.1 Definition 3. 1. An m×n Matrix is a rectangular array of entries (numbers, variables, functions, etc) with m rows and n columns. The entries are called elements.   a11 a12 · · · a1n  a21 a22 · · · a2n    A =  .. ..   . .  am1 am2 · · · amn 2. The matrix is said to be square if m = n. For a square matrix the main diagonal is the entries a11 , a22 , · · · , ann and the sum of the main diagonal elements is called the trace of A. 3. A matrix with n rows and 1 column is called a column vector and a matrix with n columns and 1 row is called a row vector. 4. Two m×n matrices A and B are said to be equal if and only if aij = bij for all i = 1, · · · , m and all j = 1, · · · , n, i.e., two matrices of the same size are equal if and only if every corresponding entry is equal. 5. We denote the set of all m × n matrices by M (m, n) and we define an addition for A, B ∈ M (m, n) by       a11 a12 · · · a1n b11 b12 · · · b1n a11 + b11 a12 + b12 · · · a1n + b1n  a21 a22 · · · a2n   b21 b22 · · · b2n   a21 + b21 a22 + b22 · · · a2n + b2n         .. . .. + .. ..  =  .. ..  .  .   . .   . . am1 am2 · · · amn bm1 bm2 · · · bmn am1 + bm1 am2 + bm2 · · · amn + bmn That is, the i, j-th entry in the sum of two matrices is matrix. 6. We also define a scalar multiplication on M (m, n) by  ka11 ka12 · · ·  ka21 ka22 · · ·  kA =  ..  . kam1 kam2 · · ·

the sum of the i, j-th entries of each

 ka1n ka2n   ..  . .  kamn

7. The matrix with all zero entries is called the Zero Matrix and is denoted by O. 8. Properties of Matrix Addition and Scalar Multiplication Let A, B and C be in M (m, n) and k1 , k2 be any scalars. Then (a) (commutative property) (A + B) = (B + A). (b) (distributive law 1) k1 (A + B) = (k1 A + k1 B). (c) (distributive law 2) (k1 + k2 )A = k1 A + k2 A. (d) (distributive law 3) k1 (k2 A) = (k1 k2 )A. (e) (additive identity) A + O = O + A = A. (f) (additive inverse) A + (−A) = O. (g) (multiplicative identity) 1A = A. Therefore we see that M (m, n) is a vector space. The standard basis vectors consist of the mnmatrices Eij whose ij-entry is a 1 and all other entries are 0. Thus we see that the dimension of M (m, n) is mn (m times n).

3

Definition 4 (Matrix Multiplication). One more very important operation that can be performed with matrices of appropriate sizes is a matrix multiplication. BUT in order for a pair of matrices A and B be conformable for the multiplication AB it must be that the number of columns of A must equal the number of rows of B. Notice this means that it is entirely possible that we can multiply A times B but we cannot multiply B times A. In order to describe the operation of A times B let us assume that A ∈ M (m, s) and B ∈ M (s, n) so that the compatibility condition is satisfied. Then the matrix C = AB will be in M (m, n) and all we need to do is tell you the value of Cij for i = 1, · · · , m and j = 1, · · · , n in order to define the matrix multiplication: s X Ci,j = ai` b`j . `=1

This multiplication can be interpreted in terms of inner products of vectors as follows. First we notice that a matrix can be viewed in two useful ways. On the one hand it can be viewed as a column of row vectors   A1  A2      A =  ..  where Ai = ai1 ai2 · · · ain ∈ Rn .  .  Am On the other hand it can be viewed as a row of column vectors   A = A1 A2 · · ·

An

 a1j  a2j    where Aj =  ..  Rm .  . 



amj With this notation we can see that the ij-th entry of C = AB is nothing more than the dot (or inner) product of the i-th row of A with the j-th column of B.

Properties of Matrix Multiplication (Assuming all matrices below are conformable for multiplication) 1. Even if A and B are square matrices, in general AB 6= BA. 2. A(BC) = (AB)C. 3. A(B + C) = (AB) + (AC). 4. (B + C)A = (BA) + (CA). Another useful operation on matrices is the transpose of a matrix A defined by   a11 a21 · · · am1  a12 a22 · · · am2    AT =  .. ..   . .  a1n a2n · · · amn Thus we see that AT ∈ M (n, m) is the matrix obtained from A by interchanging the rows and columns. Therefore if we have a column vector x then xT is a row vector. Properties of Matrix Transpose (Assuming all matrices below are conformable for multiplication) 4

1. 2. 3. 4.

(AT )T = A. (A + B)T = AT + B T . (AB)T = B T AT . (kA)T = kAT .

Definition 5 (Other Special Matrices). Other special square matrices include triangular matrices and the special case of diagonal matrices. Another special type of matrix is a triangular matrix. They can be either upper or lower triangular. An upper triangular matrix is a matrix with all zeros below the main diagonal and a lower triangular matrix is a matrix with all zeros above the main diagonal. A matrix is said to be a diagonal matrix is all the entries off the main diagonal are zeros. The n × n diagonal matrix with all ones on the diagonal is called the identity matrix denoted In . The identity matrix satisfies properties in common with the number one. Namely, for any n × n matrix A we have AIn = In A = A. An n × n matrix A is said to by Symmetric if AT = A. In orther words aij = aji for all i and j.

Linear Systems of Equations

Section 8.2

A linear system of m equations in n unknowns (with unknowns denoted by {xj }nj=1 is a11 x1 + a12 x2 + · · · + a1n xn = b1 a21 x1 + a22 x2 + · · · + a2n xn = b2 .. . am1 x1 + am2 x2 + · · · + amn xn = bm

(1)

The quantities aij are called the coefficients of the system and are given constants. The system is said to be homogeneous if all the bj are zero and non-homogeneous otherwise. A solution is a set of values for the unknowns {xj }nj=1 which when substituted into the system of equations renders the equations valid, i.e. the equations are satisfied. In order to solve a linear system there are three allowable operations called elementary operations which can be preformed on the system to obtain and equivalent system, i.e., a system with exactly the same solutions. 1. Interchange the position of the equations. 2. Multiply any equation by a nonzero constant. 3. Add a nonzero multiple of one equation to another equation. The Augmented matrix of the system (1)  a11  a21   ..  . am1

is a12 a22

··· ···

am2 · · ·

a1n a2n .. .

b1 b2 .. .

   . 

amn bm

The analog of the elementary operations on equations for the augmented matrix is the Elementary Row Operations

5

1. Interchange the position of the rows. 2. Multiply any row by a nonzero constant. 3. Add a nonzero multiple of one row to anothe rrow. In the method known as Gaussian Elimination the row operations are applied until one obtains the Row-Echelon Form: 1. The first nonzero element in every nonzero row is a 1 (this position is called a pivot). 2. In consecutive nonzero rows, the first entry 1 in the lower row appears to the right of the 1 in the row above it. 3. Rows with all zeros are at the bottom of the matrix. For the Gauss-Jordan Method one continues using the row operations to obtain the Reduced Row-Echelon Form (sometimes called the Row Reduced Echelon Form or RREF) : 1. The first nonzero element in every nonzero row is a 1 (this position is called a pivot). 2. In consecutive nonzero rows, the first entry 1 in the lower row appears to the right of the 1 in the row above it. 3. Rows with all zeros are at the bottom of the matrix. 4. Above any pivot there are all zeros. A System of equations is said to be consistent if it has at least one solution. If it has no solutions then it is called inconsistent. If a system is consistent then there are two possibilities: 1. There is a unique solution, i.e. there is only one solution. 2. There are infinitely many solutions. A homogeneous system of equations is always consistent and a homogeneous m × n system with m < n always has infinitely many solutions.

Rank of a Matrix and Linear Systems of Equations

Section 8.3

Using the definition of matrix multiplication and equality of matrices we can write the system of equations (1) in the form:       a11 a12 · · · a1n x1 b1  a21 a22 · · · a2n   x2   b2        AX = B, where A =  .. ..  , X =  ..  , B =  ..  .  . .  .  .  am1 am2 · · · amn xn bm In order to exploit this formulation of a linear system we will introduce several important tools that can be of use in practice and in a theoretical study. Definition 6 (Rank of a Matrix). The rank of an m × n matrix A is the number of linearly independent row vectors in A. Definition 7 (Row and Column Space). 1. The rows of A which we denote by {Ai }m i=1 span a subspace of Rn called the Row Space of A denoted by RA . 6

2. The columns of A which we denote by {Aj }nj=1 span a subspace of Rm called the Column Space of A denoted by CA . We have the following result which is useful for finding the rank of a matrix. If B is a row-echelon form of A, then 1. {the row space of A } = {the row space of B } . 2. The nonzero rows of B form a basis for RA . 3. Rank(A) = the number of nonzero rows of B. The following are all the same 1. The rank of A, i.e. number of linearly independent rows of A. 2. The dimension of RA , i.e. number of elements in a basis for RA . 3. The number of linearly independent columns of A. 4. The dimension of CA . A linear system AX = B is consistent if and only if the rank of A is the same as the rank of the augmented matrix (A|B). If a system is consistent and has infinitely many solutions then the solution will contain a number of arbitrary parameters. In particular for an m × n system if the rank of A is r then the number of free parameters is n − r.

Determinant of a Square Matrix

Section 8.5

When m = n we obtain a square matrix and associate to it a number called the determinant   a11 a12 · · · a1n a11 a12 · · · a1n a21 a22 · · · a2n  a21 a22 · · · a2n    A =  .. .. . ..  , det(A) = ..  .  . . . an1 an2 · · · ann an1 an2 · · · ann The determinant for an n × n matrix is defined by first defining the determinant of 1×, 2 × 2 and 3 × 3. Then we introduce the method of expansion by minors and cofactors which allows us to easily define the determinant of any n × n matrix. 1. If A = (a11 ) then |A| = a11 .   a11 a12 2. If A = then |A| = a11 a22 − a12 a21 . a21 a22 Now given an n × n matrix A we define a reduction process for finding the determinant called expansion by minors and cofactors. Definition 8. Given an n × n matrix A we define the Minors and Cofactors as follows: 1. The ijth Minor Mij is the determinant of the (n − 1) × (n − 1) matrix obtained from A by deleting the ith row and the jth column. 2. The ijth Cofactor Cij is a signed minor. We define Cij = (−1)i+j Mij .

7

Definition 9. Given an n × n matrix A we define the determinant expanded by the i row by: det(A) = ai1 Ci1 + ai2 Ci2 + · · · + ain Cin =

n X

aij Cij .

j=1

Given an n × n matrix A we define the determinant expanded by the j column by: det(A) = a1j C1j + a2j C2j + · · · + anj Cnj =

n X

aij Cij .

i=1

  a11 a12 a13 Example 5. Consider a 3 × 3 matrix A = a21 a22 a23 . The determinant expanded by the first a31 a32 a33 row is det(A) = a11 C11 + a12 C12 + a13 C13   a22 a23 a21 a23 a21 a22 + a12 − = a11 a31 a33 + a13 a31 a32 a32 a33 = a11 (a22 a33 − a23 a32 ) − a12 (a21 a33 − a23 a31 ) + a13 (a21 a32 − a22 a31 ) Remark 2. A determinant can be expanded along any row or column

Properties of Determinants

Section 8.5

Given an n × n matrix A 1. det(AT ) = det(A). 2. If B is an n × n matrix then det(AB) = det(A) det(B). 3. If two rows (or columns) of A are equal then det(A) = 0. 4. If a row (or column) of A has all zero entries then det(A) = 0. 5. If B is obtained from A by interchanging two rows (or columns) then det(B) = − det(A). 6. If B is obtained from A by multiplying a row (or column) by k then det(B) = k det(A). 7. If B is obtained from A by adding a any multiple of a row (or column) to another row (or column) then det(B) = det(A). 8. If A is triangular (either upper or lower) then det(A) = a11 a22 · · · ann .

8

Inverse of a Square Matrix

Section 8.6

In this section we consider trying to find the analog of multiplicative inverse of a number, i.e., a−1 , for matrices. We know that for a nonzero number a there is a number a−1 satisfying aa−1 = a−1 a = 1. Given an n × n matrix A we want to consider finding a matrix B so that BA = AB = In . Definition 10. Let A be an n × n matrix. If there exists a matrix B so that BA = AB = In then we say that A is Nonsingular (or Invertible). The matrix B is called the Inverse of A. We usually denote it by A−1 . An n × n matrix A is invertible if and only if det(A) 6= 0 Indeed we can write a formula for the inverse of a matrix using the so called Adjoint matrix. Definition 11. Let A be an n × n matrix. The Adjoint matrix adj(A) is the transpose of the matrix of cofactors of A  T C11 C12 · · · C1n  C21 C22 · · · C2n    adj(A) =  .. ..  ,  . .  Cn1 Cn2 · · · Cnn

A

−1

 =

1 det(A)

 adj(A)

Best method for finding the inverse is to write the augmented matrix (A|In ) and use elementary row operations to reduce the left side to In . In so doing the right side is transformed into A−1 . Important consequences of these results are Given an n × n matrix A 1. AX = 0 has only the trivial solution, i.e. X = 0, if and only if A is nonsingular. 2. AX = 0 has a nontrivial solution if and only if det(A) = 0.

The Eigenvalue Problem

Section 8.8

Given an n × n matrix A a number λ is said to be an eigenvalue of A if there exists a nonzero solution of the V of the linear system AV = λV . The vector V is called a eigenvector corresponding to λ. Form what we learned in the previous sections the linear system AV = λV which can be written as the homogeneous linear system (λIn − A)V = 0. Given an n × n matrix A a number λ is an eigenvalue if and only if it is a root of the polynomial equation det(λIn − A) = 0.

9