Idempotency and Projection Matrices
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
1 / 32
A square matrix P is idempotent iff PP = P.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
2 / 32
A square matrix P is a projection matrix that projects onto the vector space S ⊆ Rn iff (a) P is idempotent, (b) Px ∈ S ∀ x ∈ Rn , and (c) Pz = z ∀ z ∈ S.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
3 / 32
Result P.1:
Suppose P is an idempotent matrix. Prove that P projects onto a vector space S iff S = C(P).
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
4 / 32
Proof of Result P.1:
(=⇒) Property (b) of a projection matrix implies that Px ∈ S ∀ x ∴ C(P) ⊆ S.
By Property (c) of a projection matrix, Pz = z ∀ z ∈ S. Thus, any z ∈ S also in C(P). ∴ S ⊆ C(P), and we have C(P) = S.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
5 / 32
(⇐=) Need to show that any idempotent P is a projection matrix that projects onto C(P) as follows: (a) PP = P, (b) Px ∈ C(P) ∀ x, (c) z ∈ C(P) ⇒ ∃ x 3 z = Px. Therefore, Pz = PPx = Px = z.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
6 / 32
Result A.14:
AA− is a projection matrix that projects onto C(A).
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
7 / 32
Proof of Result A.14:
(a) (AA− )(AA− ) = (AA− A)A− = AA− . Therefore, AA− is idempotent. (b) AA− x = Az ∀ x, where z = A− x. Thus AA− x ∈ C(A) ∀ x. (c) ∀ z ∈ C(A), ∃ y 3 z = Ay, ∴ AA− z = AA− Ay = Ay = z.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
8 / 32
Alternatively, we could have proved idempotency and then shown C(A) = C(AA− ) as below: Ax = (AA− A)x = (AA− )Ax ⇒ C(A) ⊆ C(AA− ). AA− x = A(A− x) ⇒ C(AA− ) ⊆ C(A). ∴ C(A) = C(AA− ).
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
9 / 32
Result A.15:
I − A− A is a projection matrix that projects onto N (A).
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
10 / 32
Proof of Result A.15:
(a) (I − A− A)(I − A− A) = I − A− A − A− A + A− AA− A = I − A− A − A− A + A− A = I − A− A.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
11 / 32
(b) Note that A(I − A− A)x = (A − AA− A)x
c Copyright 2012 Dan Nettleton (Iowa State University)
= (A − A)x = 0 ∀ x. ∴ (I − A− A)x ∈ N (A) ∀ x.
Statistics 611
12 / 32
(c) If z ∈ N (A), then (I − A− A)z = z − A− Az =z−0 = z.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
13 / 32
Prove that C(I − A− A) = N (A).
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
14 / 32
Proof:
The result follows from Result A.15 and P.1. An alternative proof is as follows.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
15 / 32
Proof: Suppose z ∈ N (A). Then Az = 0 ⇒ A− Az = 0 ⇒ z − A− Az = z ⇒ (I − A− A)z = z ⇒ z ∈ C(I − A− A).
∴ N (A) ⊆ C(I − A− A).
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
16 / 32
Suppose z ∈ C(I − A− A). Then ∃ x 3 z = (I − A− A)x. Thus Az = A(I − A− A)x = (A − AA− A)x = (A − A)x = 0. Thus, z ∈ N (A). It follows that C(I − A− A) ⊆ N (A). Hence, C(I − A− A) = N (A).
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
17 / 32
Result A.16:
Any symmetric and idempotent matrix P is the unique symmetric projection matrix that projects onto C(P).
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
18 / 32
Proof of Result A.16:
Suppose Q is a symmetric projection matrix that projects onto C(P). Then Pz = Qz = z ∀ z ∈ C(P) ⇒ PPx = QPx ∀ x ⇒ Px = QPx ∀ x ⇒ P = QP.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
19 / 32
Now Q is a projection matrix that projects on C(P), therefore, C(P) = C(Q). Thus Qz = Pz = z ∀ z ∈ C(Q) ⇒ QQx = PQx ∀ x ⇒ Qx = PQx ∀ x ⇒ Q = PQ.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
20 / 32
Now note that (P − Q)0 (P − Q) = P0 P − P0 Q − Q0 P + Q0 Q = PP − PQ − QP + QQ =P−Q−P+Q = 0. ∴ P − Q = 0 ⇒ P = Q.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
21 / 32
Any symmetric, idempotent matrix P is known as an orthogonal projection matrix because (Px) ⊥ (x − Px), i.e., (Px)0 (x − Px) = x0 Px − x0 P0 Px
c Copyright 2012 Dan Nettleton (Iowa State University)
= x0 Px − x0 PPx = x0 Px − x0 Px = 0.
Statistics 611
22 / 32
Corollary A.4:
If P is a symmetric projection matrix, then I − P is a symmetric projection matrix that projects onto C(P)⊥ = N (P).
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
23 / 32
Proof of Corollary A.4:
First note that C(P)⊥ = N (P0 ) = N (P) by the symmetry of P. We need to show that properties (a-c) of a projection matrix hold for I − P onto N (P).
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
24 / 32
(a) Is I − P idempotent? (I − P)(I − P) = I − P − P + PP
c Copyright 2012 Dan Nettleton (Iowa State University)
=I−P−P+P = I − P.
Statistics 611
25 / 32
(b) Is (I − P)x ∈ N (P) ∀ x? P(I − P)x = (P − PP)x = (P − P)x = 0. ∴ (I − P)x ∈ N (P) ∀ x.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
26 / 32
(c) Does (I − P)z = z ∀ z ∈ N (P)? ∀ z ∈ N (P), (I − P)z = z − Pz =z−0 = z. Finally, we should note that (I − P)0 = I0 − P0 = I − P so that I − P is symmetric as claimed in statement of the result.
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
27 / 32
Suppose A =
" # 1
.
1
Find the orthogonal projection matrix that projects onto C(A). Find the orthogonal projection matrix that projects onto N (A0 ).
Find the orthogonal projection of x =
" # 4 2
onto C(A) and onto
N (A0 ).
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
28 / 32
Need to find a symmetric, idempotent matrix whose column space is C(A), where C(A) = {x ∈ R2 : x1 = x2 }. Thus, P must have the form
c Copyright 2012 Dan Nettleton (Iowa State University)
P=
" # a a
.
a a
Statistics 611
29 / 32
Because P must be idempotent, "
a a a a
This implies
2a2
#" # a a a a
" =
2a2 2a2 2a2 2a2
= a ⇒ a = 1/2. ∴ P =
c Copyright 2012 Dan Nettleton (Iowa State University)
#
" =
a a
# .
a a
" # 1/2 1/2
.
1/2 1/2
Statistics 611
30 / 32
We know " I−P=
1/2 −1/2 −1/2
#
1/2
is the orthogonal projection matrix that projects onto C(P)⊥ = C(A)⊥ = N (A0 ).
c Copyright 2012 Dan Nettleton (Iowa State University)
Statistics 611
31 / 32
P
" # 4 2
=
" # 3 3
, (I − P)
" # 4
" =
2
# 1 −1
.
C(A) (3,3) (4, 2)
(1,-1)
c Copyright 2012 Dan Nettleton (Iowa State University)
N(A')
Statistics 611
32 / 32
