INNER PRODUCTS

1. Real Inner Products Definition 1. An inner product on a real vector space V is a function (u, v) 7→ hu, vi from V × V to R satisfying (1) hαu + βv, wi = α hu, wi + β hv, si for all u, v, w ∈ V and all α, β ∈ R; (2) hu, vi = hv, ui for all u, v ∈ V ; (3) hv, vi ≥ 0 for all v ∈ V , with equality if and only if v = 0. A real vector space together with an inner product is called a real inner product space. Item 1 asserts that the map (u, v) 7→ hu, vi is linear in the first variable. In view of 2, it is also linear in the second variable. Example 2. The usual dot product hu, vi = u · v =

n X

ui vi

i=1

defines an inner product on Rn . Example 3. The set C([a, b]) of all continuous real valued functions on the interval [0, 1] forms a real vector space under pointwise addition and scaling. We can define an inner product on C([a, b]) by Z b hf, gi = f (x)g(x) dx. a

Definition 4. Let V be a real vector space with inner product h·, ·i. For any v ∈ V we define the norm of v by 1/2 kvk = hv, vi . Note that the square root on the right is defined as a non-negative real number by 3. Lemma 5. Let V be a real vector space with inner product h·, ·i. (1) (Positivity) kvk ≥ 0, with equality if and only if v = 0. (2) (Homogeneity) kαvk = |α| kvk for every α ∈ R and every v ∈ V . (3) (Cauchy-Schwarz Inequality) | hu, vi | ≤ kuk kvk, with equality if and only if u and v are parallel. (4) (Triangle Inequality) ku + vk ≤ kuk + kvk. Proof. Positivity is immediate from item 3 in the definition of inner product. For homogeneity, we have kαvk2 = hαv, αvi = α2 hv, vi = α2 kvk2 . The result follows by taking square roots. 1

INNER PRODUCTS

2

We now turn to the Cauchy-Schwarz Inequality. If u and v are parallel, then one is a scalar multiple of the other, say v = αu for some α ∈ R. Therefore | hu, vi | = | hu, αui | = |α hu, ui | = |α| kuk2 = kuk kαuk = kuk kvk. On the other hand, if u and v are not parallel, then u + αv 6= 0 for every α ∈ R, so, by positivity, 0 < ku + αvk2 = hu + αv, u + αvi = kuk2 + 2 hu, vi α + kvk2 α2 In particular, taking α = − hu,vi kvk2 gives 2

0 < kuk2 −

hu, vi , kvk2

from which the Cauchy-Schwarz Inequality follows. Finally, we turn to the proof of the Triangle Inequality. We have ku + vk2 = hu + v, u + vi = kuk2 + 2 hu, vi + kvk2 ≤ kuk2 + 2| hu, vi | + kvk2 ≤ kuk2 + kuk kvk + kvk2 2

= (kuk + kvk) . Taking square roots gives the desired result.



Definition 6. Two vectors u and v in a real inner product space are said to be orthogonal if hu, vi = 0. Theorem 7 (Pythagorean Theorem). If u and v are orthogonal vectors in a real inner product space, then ku + vk2 = kuk2 + kvk2 . Proof. ku + vk2 = hu + v, u + vi = kuk2 + 2 hu, vi + kvk2 = kuk2 + kvk2 .  1.1. Exercises. (1) When does equality hold in the Triangle Inequality? (2) For any non-negative integer n, define cn , sn ∈ C([0, 2π]) by cn (x) = cos nx,

sn (x) = sin nx.

Using the inner product of Example 3: (a) Calculate kcn k and ksn k. (b) Show that cn and sm are orthogonal for every m, n ∈ Z+ . (c) Show that cn is orthogonal to cm and sn is orthogonal to xm whenever n 6= m. (3) Prove the converse of the Pythagorean Theorem: In a real inner product space, if ku + vk2 = kuk2 + kvk2 , then u and v are orthogonal.

INNER PRODUCTS

3

2. The Complex Case It is tempting to define an inner product on a complex vector space by simply transcribing the definition of a real inner product, but allowing the scalars to be complex. Unfortunately, this does not lead to anything useful: There would be no inner products on any complex vector space containing a non-zero vector (see Exercise 1). A slight refinement of the second item in the definition is needed. Definition 8. An inner product on a complex vector space V is a function (u, v) 7→ hu, vi from V × V to C satisfying (1) hαu + βv, wi = α hu, wi + β hv, si for all u, v, w ∈ V and all α, β ∈ C; (2) hu, vi = hv, ui for all u, v ∈ V ; (3) hv, vi ≥ 0 for all v ∈ V , with equality if and only if v = 0. A complex vector space together with an inner product is called a complex inner product space. We will use to phrase inner product space to refer to either a real or complex inner product space. Note that a complex inner product is linear in the first variable, but not in the second. In fact, we have hu, αv + βwi = hαv + βw, ui = α hv, ui + β hw, ui = αhv, ui + βhw, ui = α hu, vi + β hu, wi . We say that h·, ·i is conjugate linear in the second variable. Example 9. The Hermitian inner product on Cn is defined by hz, wi =

n X

zi wi .

i=1 1/2

As in the real case, we define kvk = hv, vi , and we say u and v are orthogonal if hu, vi = 0. Complex variants of Lemma 5 and Theorem 7 can be established as in the real case. We state them here for completeness. The proofs are left to the exercises. Lemma 10. Let V be a complex vector space with inner product h·, ·i. (1) (Positivity) kvk ≥ 0, with equality if and only if v = 0. (2) (Homogeneity) kαvk = |α| kvk for every α ∈ C and every v ∈ V . (3) (Cauchy-Schwarz Inequality) | hu, vi | ≤ kuk kvk, with equality if and only if u and v are parallel. (4) (Triangle Inequality) ku + vk ≤ kuk + kvk. Theorem 11 (Pythagorean Theorem). If u and v are orthogonal vectors in a complex inner product space, then ku + vk2 = kuk2 + kvk2 .

INNER PRODUCTS

4

2.1. Exercises. (1) Show that if the conjugation on the right side of 2 were omitted no inner products would exist on any complex vector space containing a non-zero vector. (2) Let V be a complex vector space. We may view V as a real vector space by simply ignoring non-real scalars. Now suppose that h·, ·i is a complex inner product on V , and define hhu, vii = Re hu, vi . (a) Show that this defines a real inner product. (b) Show that the real inner product hh·, ·ii and the complex inner product h·, ·i define the same norm. (c) Show that orthogonality with respect to the complex inner product implies orthogonality with respect to the real inner product, but not conversely. It follows that the real Pythagorean Theorem is stronger than the complex version. (3) Let h·, ·i be an inner product on a complex vector space V . (a) Show that for every u, v ∈ V and every α ∈ C we have ku + αvk = kuk2 + 2 Re (α hu, vi) + |α|2 kvk2 (b) Prove the Cauchy-Schwarz Inequality for complex inner products. Hint: Take α = hu,vi kvk2 . 3. Orthonormal Sets Definition 12. A set of vectors {e1 , . . . , en } in an inner product space is said to be orthogonal if hei , ej i = 0 whenever i 6= j. If, in addition, kei k = 1 for each i, we call the set orthonormal. An orthonormal set which is also a basis for V is an orthonormal basis. Theorem 13. If {e1 , . . . , en } is an orthonormal basis for V , then for every v ∈ V we have n X v= hv, ei i ei . i=1

Corollary 14. If {e1 , . . . , en } is an orthonormal basis for V , then for any u, v ∈ V we have n X hu, vi = hu, ei i hv, ei i. i=1

Note that the corollary asserts that the inner product of two vectors in a real or complex finite dimensional inner product space is the Euclidean or, respectively, Hermitian inner product of their coordinate vectors in Rn or Cn . Corollary 15. Let T be a linear transformation from a finite dimensional inner product space V to a finite dimensional inner product space W . If A = [aij ] is the matrix of T with respect to ordered orthonormal bases (e1 , . . . , en ) and (f1 , . . . , fm ), respectively, then aij = hT ej , ei i . Proof. The jth column of is the coordinate vector of T ej with respect to the basis (f1 , . . . , fm ). The result follows from Corollary 13. 

INNER PRODUCTS

5

Theorem 16 (Bessel’s Inequality). If {e1 , . . . , en } is orthonormal in V , then for every v ∈ V we have n X | hv, ei i |2 ≤ kvk2 . i=1

Moreover, we have equality for every v ∈ V if and only if the set is an orthonormal basis. Theorem 17. Let {v1 , . . . , vn } be linearly independent. There is an orthonormal set {e1 , . . . , en } such that for 1 ≤ k ≤ n we have span{e1 , . . . , ek } = span{v1 , . . . , vk }. Proof (Gram-Schmidt Process): Let Vk = span{v1 , . . . , vk }. We first construct, by induction on k, an orthogonal set {w1 , . . . , wn } such that span{w1 , . . . , wk } = Vk . wk . The required orthonormal set is then obtained by setting ek = kw kk We begin by setting w1 = v1 . For 1 < k ≤ n, suppose by induction that we have an orthogonal set {w1 , . . . , wk−1 } which spans Vk−1 . Since dim Vk−1 = k − 1, it follows that {w1 , . . . , wk−1 } is a basis for Vk−1 , and in particular, wi 6= 0 for 1 ≤ i ≤ k − 1. Let k−1 X hvk , wi i wk = vk − wi . kwi k2 i=1 It follows from the orthogonality of {w1 , . . . , wk−1 } that hwk , wi i = 0 for 1 ≤ i ≤ k − 1, and further that wk 6= 0, since vk 6∈ Vk−1 . Moreover, wk ∈ Vk , so {w1 , . . . , wk } is a linearly independent set in Vk . Since dim Vk = k, it must be a basis for Vk . This completes the proof.  Corollary 18. Let V be an inner product space of finite dimension n, and let {e1 , . . . , ek } be orthonormal in V . There are ek+1 , . . . , en ∈ V such that {e1 , . . . , en } is an orthonormal basis. In particular, every finite dimensional inner product space has an orthonormal basis. 4. Orthogonal Projections Definition 19. Let V be an inner product space and let E ⊂ V . The orthogonal complement of E is E ⊥ = {v ∈ V : hv, wi = 0 for every w ∈ E}. Lemma 20. E ⊥ is a subspace of V . Theorem 21 (Projection Theorem). Let V be an inner product space and let W be a finite dimensional subspace. For every v ∈ V there are unique v1 , v2 ∈ V such that (1) v1 ∈ W and v2 ∈ W ⊥ ; (2) v = v1 + v2 . Proof. Let {e1 , . . . , ek } be an orthonormal basis for W . If v1 and v2 are as advertised, then hv, ei i = hv1 + v2 , ei i = hv1 , ei i + hv2 , ei i = hv1 , ei i so k k X X hv, ei i ei = hv1 , ei i ei = v1 i=1

i=1

INNER PRODUCTS

6

so v1 , and hence also v2 , is uniquely determined. To complete the proof, let v ∈ V , and define (1)

k X

v1 =

hv, ei i ei

i=1

and v2 = v − v1 . Clearly v1 ∈ W and v = v1 + v2 . It only remains to check that v2 ∈ W ⊥ . It is straightforward to check that v2 is orthogonal to each ei , and therefore to any linear combination of {e1 , . . . , ek }. Since {e1 , . . . , ek } is a basis for W , it follows that v2 is orthogonal to each member of W , which completes the proof.  Remark 22. The vector v1 in the Projection Theorem is the orthogonal projection of v on the subspace W . The proof shows that v1 is given by (1), where {e1 , . . . , ek } can be any orthonormal basis for the subspace W . The linear operator defined by (2)

Pv =

k X

hv, ei i ei

i=1

is the orthogonal projector of V onto W . The uniqueness assertion in the Projection Theorem assures us that the operator P defined by (2) does not depend on the choice of an orthonormal basis for W . Corollary 23. If W is a subspace of a finite dimensional inner product space V , then dim W + dim W ⊥ = dim V. Proof. If {e1 , . . . , em } is an orthonormal basis for W and {f1 , . . . , fn } is an orthonormal basis for W ⊥ , then {e1 , . . . , em , f1 , . . . , fn } is an orthonormal set which spans V , and is therefore an orthonormal basis for V .  Definition 24. A linear functional on a real (or complex) vector space V is a linear transformation from V into R (or C). Theorem 25 (Representation Theorem for Linear Functionals). Let λ be a linear functional on a finite dimensional inner product space V . There is a unique w ∈ V such that λ(v) = hv, wi for every v ∈ V . Proof. We first establish uniqueness. Suppose that hv, w1 i = hv, w2 i for every v ∈ V . Then hv, w1 − w2 i = 0 for every v ∈ V . In particular, taking v = w1 − w2 we obtain kw1 − w2 k2 = 0, and hence w1 = w2 , establishing uniqueness. For existence, since V is finite dimensional, it has an orthonormal basis {e1 , . . . , en }. For every v ∈ V we have n X v= hv, ei i ei i=1

so Λv =

n X i=1

* hv, ei i Λei =

n X v, (Λei )ei i=1

+ = hv, wi

INNER PRODUCTS

with w=

7

n X (Λei )ei . i=1

 5. Adjoints Let V and W be finite dimensional inner product spaces over the same scalar field (R or C), and let T : V → W be linear. Our immediate goal is to define a transposed transformation T ∗ going in the reverse direction, from W to V . Lemma 26. Let V , W , and T be as above. For each w ∈ W there is a unique w∗ ∈ V such that for every v ∈ V we have hT v, wi = hv, w∗ i . Proof. Apply the Representation Theorem to the linear functional λ(v) = hT v, wi .  Definition 27. With V , W , and T as above, we define the adjoint mapping T ∗ : W → V by defining T ∗ w to be the unique w∗ ∈ V such that hT v, wi = hv, w∗ i for every v ∈ V . Thus the map T ∗ is characterized by the identity hT v, wi = hv, T ∗ wi for all v ∈ V and w ∈ W . Lemma 28. T ∗ is linear. Proof. Let w1 , w2 ∈ W and let α1 , α2 be scalars. Then for every v ∈ V we have hv, T ∗ (α1 w1 + α2 w2 )i = hT v, α1 w1 + α2 w2 i = α1 hT v, w1 i + α2 hT v, w2 i = α1 hv, T ∗ w1 i + α2 hv, T ∗ w2 i = hv, α1 T ∗ w1 + α2 T ∗ w2 i Since this holds for every v ∈ V , it follows that T ∗ (α1 w1 + α2 w2 ) = α1 T ∗ w1 + α2 T ∗ w2 .  Theorem 29. (1) (T ∗ )∗ = T (2) (S + T )∗ = S ∗ + T ∗ (3) (αT )∗ = αT ∗ (4) (ST )∗ = T ∗ S ∗ Theorem 30. Let V and W be finite dimensional inner product spaces and let T be a linear transformation from V to W . Let A be the matrix of T with respect to T ordered orthonormal bases. Then the matrix of T ∗ is A . This is an immediate consequence of Corollary 15 and the definition of the adjoint transformation. Of course, in the real case, the bar over A can be omitted.

INNER PRODUCTS

8

Definition 31. A linear operator T on a finite dimensional inner product space is self-adjoint if T ∗ = T . A real n × n matrix that defines a self adjoint operator on Rn is called real symmetric. A complex n × n matrix that defines a self-adjoint operator on Cn is Hermitian. Note that a real n×n matrix is real symmetric if and only if AT = A. A complex T n × n matrix is Hermitian if and only if A = A. Lemma 32. All eigenvalues of a self-adjoint operator are real. Proof. Let α be an eigenvalue of a self-adjoint operator with eigenvector v. Then αkvk2 = α hv, vi = hαv, vi = hT v, vi = hv, T vi = hv, αvi = αkvk2 so α = α.



Note that we do not (yet) assert that eigenvalues exist. Corollary 33. Every Hermitian matrix has a real eigenvalue. Proof. An n × n Hermitian matrix defines a self adjoint operator on Cn , which has a complex eigenvalue by the Fundamental Theorem of Algebra. By the previous lemma, this complex eigenvalue must be real.  Corollary 34. Every self-adjoint operator has a real eigenvalue. Proof. In the complex case, this is immediate from Lemma 32 and the Fundamental Theorem of Algebra. For the real case, it suffices to show that the matrix A of the transformation with respect to some orthonormal basis has a real eigenvalue. But the matrix is real symmetric, and hence also Hermitian, so, by the previous corollary, it has a real eigenvalue.  6. Orthogonal and Unitary Operators Theorem 35 (Polarization Identities). (1) If V is a real inner product space, then for every u, v ∈ V we have
1 hu, vi = ku + vk2 − kuk2 − kvk2 . 2 (2) If V is a complex inner product space, then for every u, v ∈ V we have
1 ku + vk2 + iku + ivk2 − ku − vk2 − iku − ivk2 . hu, vi = 4 Proof. For any scalar α we have (3)

ku + αvk2 = hu + αv, u + αvi = kuk2 + α hu, vi + α hv, ui + |α|2 kvk2

In the real case, we have hv, ui = hu, vi, and the result follows by taking α = 1. For the complex case, note that when |α| = 1, (3) becomes, upon multiplication by α αku + αvk2 = αkuk2 + hu, vi + α2 hu, vi + αkvk2 . In particular, taking α = 1, −1, i, and −i gives ku + vk2 = kuk2 + hu, vi + hv, ui + kvk2 −ku + vk2 = −kuk2 − hu, vi + hv, ui − kvk2 iku + ivk2 = ikuk2 + hu, vi − hv, ui + ikvk2 −iku − ivk2 = −ikuk2 + hu, vi − hv, ui − ikvk2

INNER PRODUCTS

Adding the four identities above gives the desired result.

9



Theorem 36. Let T be a linear operator on a finite dimensional vector space V . The following are equivalent. (1) For every u, v ∈ V we have hT u, T vi = hu, vi. (2) For every v ∈ V we have kT vk = kvk. (3) For every orthonormal basis {e1 , . . . , en } for V , the set {T e1 , . . . , T en } is also an orthonormal basis for V . (4) For some orthonormal basis {e1 , . . . , en } for V , the set {T e1 , . . . , T en } is also an orthonormal basis for V . (5) T ∗ T is the identity operator on V . (6) T T ∗ is the identity operator on V . Proof. The implication 1 =⇒ 2 is trivial, and the reverse implication is immediate from the Polarization Identity. The implications 1 =⇒ 3 =⇒ 4 are also trivial. 4 =⇒ 1: Let {e1 , . . . , en } be an orthonormal basis. For u, v ∈ V we can write u = α1 e1 + · · · + αn en v = β1 e1 + · · · + βn en so T u = α1 T e1 + · · · + αn T en T v = β1 T e1 + · · · + βn T en . Since {T e1 , . . . , T en } is an orthonormal basis, we have X hT u, T vi = αi β i = hu, vi . We have now established the equivalence of 1–2. The equivalence of 5 and 6 is a consequence of the general fact that one sided inverses for linear operators on finite dimensional vector spaces are two sided inverses. We complete the proof by establishing equivalence of 5 and 1. 1 =⇒ 5: Let u ∈ V . Then for any v ∈ V we have hT ∗ T u, vi = hT u, T vi = hu, vi Since this is true for every v ∈ V , it follows that T ∗ T u = u. Since u ∈ V is arbitrary, it follows that T ∗ T is the identity operator on V . 5 =⇒ 1: If T ∗ T is the identity operator, then for any u, v ∈ V we have hT u, T vi = hT ∗ T u, vi = hu, vi .  A linear operator on a real inner product space satisfying one, and hence all, of the conditions of Theorem 36 is called orthogonal. A linear operator on a complex inner product space satisfying one, and hence all, of the conditions of Theorem 36 is called unitary.

INNER PRODUCTS

10

7. Spectral Theorem In this section, we identify the linear operators on a finite dimensional inner product space V which can be diagonalized by an orthogonal (in the real case) or unitary (in the complex case) transformation. Equivalently, we want to identify the linear the linear operators T on V such that V has an orthonormal basis consisting of eigenvectors for T . Suppose that T is a linear transformation on an inner product space V , and that {e1 , . . . , en } is an orthonormal basis for V consisting of eigenvectors of T , with corresponding eigenvalues λ1 , . . . , λn . Then hT ∗ ej , ei i = hej , T ei i = hej , λi ei i = λi δij . It follows that T ∗ ej = λj ej . Therefore, T ∗ T ej = T T ∗ ej = |λj |2 ej . Since the operators T ∗ T and T T ∗ agree on a basis, we have T ∗ T = T T ∗ . Definition 37. A linear operator T on a finite dimensional inner product space is normal if T ∗ T = T T ∗ . For example, orthogonal, unitary, symmetric, and Hermitian operators are all normal. We have established the following. Lemma 38. If T is a linear operator on an inner product space V which admits an orthonormal basis consisting of eigenvectors for T , then T is normal. The Spectral Theorem asserts that the converse holds for linear operators on complex inner product spaces. Theorem 39 (Spectral Theorem). Let T be a normal operator on a finite dimensional complex inner product space V . Then V has an orthonormal basis consisting of eigenvectors for T . The proof of the Spectral Theorem requires some preliminary lemmas. Lemma 40. If T is a normal operator on V , then kT ∗ vk = kT vk for every v ∈ V . Proof. kT vk2 = hT v, T vi = hT ∗ T v, vi = hT T ∗ v, vi = hT ∗ v, T ∗ vi = kT ∗ vk2 .  Corollary 41. If T is normal then T and T ∗ have the same null space. Corollary 42. Let T be a normal operator with eigenvalue λ, and corresponding eigenvector v. Then λ is an eigenvalue for T ∗ with the same eigenvector v. Proof. Since T is normal, so is T − λI, so T − λI and (T − λI)∗ = T ∗ − λI have the same null space.  Lemma 43. Let T be a normal operator on V with eigenvalue λ. Let W be the corresponding eigenspace. Then W ⊥ is invariant under T . Proof. Let v ∈ W ⊥ . Then for any w ∈ W we have

 hT v, wi = hv, T ∗ wi = v, λw = λ hv, wi = 0 so T v ∈ W ⊥ .



INNER PRODUCTS

11

Proof of the Spectral Theorem. We use induction on the dimension n of V . If n = 1, then any unit vector in V forms an orthonormal basis. For n > 1, suppose by induction that the conclusion holds for every complex inner product space of dimension less than n. By the Fundamental Theorem of Algebra, the characteristic polynomial of T has a complex root λ, which must be an eigenvalue of T . Let W be the corresponding eigenspace, and let {e1 , . . . , ek } be an orthonormal basis for W . By Lemma 43, V0 = W ⊥ is invariant under T , so the restriction T0 of T to V0 is a linear operator on V0 . Further, the restriction h·, ·i0 of the inner product to V0 × V0 is an inner product on V0 . One easily verifies that T0∗ is the restriction of T ∗ to V0 , and so T0 is normal. Since the dimension of V0 is less than n, it follows from our induction hypothesis that V0 has an orthonormal basis {f1 , . . . , f` } consisting of eigenvalues for T0 . It follows that {e1 , . . . , ek , f1 , . . . , f` } is an orthonormal basis for V .  7.1. The real case. The conclusion of the Spectral Theorem may fail for normal operators on real inner product spaces for the simple reason that such operators need not have eigenvalues. For example, a rotation of the plane through an angle which is not a multiple of pi is normal but has no eigenvalues. Lemma 44. Let T be a linear operator on a finite dimensional real inner product space V . If V has an orthonormal basis consisting of eigenvectors of T , then T is self-adjoint. Proof. Let {e1 , . . . , en } be an orthonormal basis of V consisting of eigenvectors for T , with corresponding eigenvalues λ1 , . . . , λn . Arguing as in the proof of Lemma 38, we obtain T ∗ ej = λn ej = T ej , so T and T ∗ agree on a bases, and are therefore equal.  The real version of the Spectral Theorem asserts the converse. Theorem 45 (Real Spectral Theorem). If T is a self adjoint operator on a finite dimensional real inner product space V , then V has an orthonormal basis consisting of eigenvectors for T . The proof of the real version of the Spectral Theorem is essentially the same as the proof of the complex version, appealing to Corollary 34 instead of the Fundamental Theorem of Algebra for the existence of eigenvalues.