The Model Model in matrix form Goodness of fit and inference

The K-Variable Linear Model Walter Sosa-Escudero

February 3, 2012

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Motivation

Consider the following natural generalization Yi = β1 + β2 X2i + . . . + βKi XKi + ui ,

i = 1, . . . , n

This the K-variable linear model K? It is as if the first variable is X1i = 1 for every i.

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Example

Yi = β1 + β2 X2i + β3i X3i + ui Yi is consumption of family i. X2i is income of family i X31 is wealth of family i.

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

The classical assumptions

1

Linearity: Yi = β1 + β2 X2i + . . . + βKi XKi + ui ,

i = 1, . . . , n.

2

Non-random X: Xki , k = 1, . . . , K are taken as deterministic, non-random variables.

3

Zero mean: E(ui ) = 0, i = 1, . . . , n.

4

Homoskedasticity: V (ui ) = σ 2 , = 1, . . . , n.

5

No serial correlation: E(ui uj ) = 0, i 6= j.

6

No multicollinearity: none of the explanatory variables can be expressed as an exact linear combination of the others.

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Interpretations As before, taking expectations E(Yi ) = β1 + β2 X2i + . . . + βKi XKi Hence

∂E(Yi ) = βk ∂Xk

Coefficients are partial derivatives. Regression as control. Replace experimentation. Common mistake: interpret as total derivatives. Example: parents education.

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Least squares estimation Let βˆk , k = 1, . . . , K be the OLS estimators. Define Yˆi ≡ βˆ1 + βˆ2 X2i + . . . + βˆKi XKi . ei ≡ Yˆi − Yi

Then the OLS esimators for βˆ1 , . . . , βˆK are the solutions to min

n X

e2i

i=1

with respecto to βˆ1 , . . . , βˆK .

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Example

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

The model in matrix notation Define the following vectors and matrices:      β1 y1   β2   y2       u= β= .  Y = .  . .   .   .  βK K×1 yn n×1    X= 

x11 x12 .. . x1n

u1 u2 .. . un



x21 . . . xK1 x22 xK2   ..  .. . .  xKn n×K

Walter Sosa-Escudero

The K-Variable Linear Model

     n×1

The Model Model in matrix form Goodness of fit and inference

The linear model Yi = β1 + β2 X2i + . . . + βKi XKi + ui ,

i = 1, . . . , n

is actually a system of n equations

Y1 = β1 + β2 X21 + . . . + βK XK1 + u1 Y2 = β1 + β2 X22 + . . . + βK XK2 + u2 ···

···

Yn = β1 + β2 X2n + . . . + βK XKn + un

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Then the  y1  ..  .   ..  . yn

linear model can be written as:    x11 x21 . . . xK1    x12 x22 xK2     =  .. ..   ..   . . .   x1n

xKn

β1 β2 .. . βK



     +   

Y = Xβ + u This is the linear model in matrix form.

Walter Sosa-Escudero



The K-Variable Linear Model

u1 .. . .. . un

     

The Model Model in matrix form Goodness of fit and inference

Basic results on matrices and random vectors Before we proceed, we need to establish some results involving matrices and vectors. Let A be a m × n matrix. A: n column vectors, or m row vectors. The column rank of A is defined as the maximum number of columns linearly dependent. Similarly, the row rank is the maximum numbers of rows that are linearly dependent. The row rank is equal to the column rank. So we will talk, in general, about the rank of a matrix A, and will denote it as ρ(A) Let A be a square (m × m) matrix. A is non singular if |A| = 6 0. In such case, there exists a unique non-singular matrix A−1 called the inverse of A such that AA−1 = A−1 A = Im . Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

A a square m × m matrix. If ρ(A) = m ⇒ |A| = 6 0 If ρ(A) < m ⇒ |A| = 0 X a n × K matrix, with ρ(X) = K (full column rank): ρ(X) = ρ(X 0 X) = k This results guarantees the existence of (X 0 X)−1 based on the rank of X.

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Let Y be a vector of K random variables:   Y1   Y =  ...  Yk    E(Y ) = µ =  

E(Y1 ) E(Y2 ) .. .

    

E(YK )

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

V (Y )

=

=

E[(Y − µ)(Y − µ)0 ]  E(Y1 − µ1 )2 E(Y1 − µ1 )(Y2 − µ2 )  E(Y2 − µ2 )2   

 ··· .. .

    E(Yk − µK )2

 =

   

V (Y1 )

Cov(Y1 , Y2 ) V (Y2 )

... ..

Cov(Y1 YK )

    

. V (YK )

The variance of a vector is called its variance-covariance matrix, an K × K matrix If V (Y ) = Σ and c is an K × 1 vector, then V (c0 Y ) = c0 V (Y )c = c0 Σc. Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Classical assumptions in matrix form

1

Linearity: Y = Xβ + u.

2

Non-random X: X is a deterministic matrix.

3

Zero mean: E(u) = 0.

4

Homoskedasticity and no serial correlation: V (u) = σ 2 In .

5

No multicollinearity: ρ(X) = K.

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

OLS estimator in matrix form

It can be show (we’ll do it later) that the OLS estimator can be expressed as: βˆ = (X 0 X)−1 X 0 Y

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Properties

2

Linearity: βˆ = AY for some matrix A. ˆ = β. Unbiasedness: E(β)

3

ˆ = σ 2 (X 0 X)−1 Variance: V (β)

1

4

Gauss-Markov Theorem: under all classical assumptions, βˆ has the minimum variance in the class of all linear and unbiased estimators.

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Proof of unbiasedness ˆ =β Unbiasedness: E(β) βˆ = (X 0 X)−1 X 0 Y = (X 0 X)−1 X 0 (Xβ + u) = β + (X 0 X)−1 X 0 u   ˆ = β + E (X 0 X)−1 X 0 u E(β) = β + (X 0 X)−1 X 0 E [u] = β

(Since E(u) = 0)

How does heteroskedasticity affect unbiasedness? Which assumptions do we use and which ones we don’t?

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Goodness of fit

It can be shown that the decomposition of squared errors still holds for the K variable model, that is X

(Yi − Y¯ )2 =

X X (Yˆi − Y¯ )2 + e2i

So our old R2 provides a goodness-of-fit measure R2 ≡

RSS ESS =1− T SS T SS

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Comments and properties on R2 . R2 ≡

ESS RSS =1− T SS T SS

0 ≤ R2 ≤ 1 (as before) βˆ maximizes R2 (why?) R2 is non-decreasing in the number of explanatory variables, K. (why?) Use and abuse of R2 .

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Simple Hypothesis

Assumption (normality): ui ∼ N (0, σ 2 )

Then, as before under all classical assumptions and when H0 : β = β0 holds t≡ q

βˆ − β0 ∼ tn−2 P S 2 / x2i

So stantard t tests are implemented as in the two variable case.

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Linear combinations of coefficients Consider the following hypothesis H0 : a1 βj + a2 βi = r a1 , a2 and r are numbers. The test for this hypothesis will be

t =

=

a1 βˆj + a2 βˆi − r q Vˆ (a1 βˆj + a2 βˆi − r) a1 βˆj + a2 βˆi − r q a21 Vˆ (βˆj ) + a22 Vˆ (βˆi ) − 2a1 a2 Cov(βˆj , βˆi )

Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Global significance Consider the hypothesis H0 : β2 = β2 = · · · βK = 0 against HA : β2 6= 0 ∨ β3 6= 0 ∨ · · · βK ∨ 0 Under the null, none of the explanatory variables account for Y . Under the alternative, at least one variable is relevant. The test statistic for this case is given by F =

ESS/(K − 1) RSS/(n − K)

which has the F (K − 1, n − K) under H0 . Idea: reject if too large. Walter Sosa-Escudero

The K-Variable Linear Model

The Model Model in matrix form Goodness of fit and inference

Alternatively, note that dividing by TSS in the numerator and denominator F =

ESS/(K − 1) R2 /(K − 1) = RSS/(n − K) (1 − R2 )/(n − K)

so the F test is checking whether R2 is significantly different from zero.

Walter Sosa-Escudero

The K-Variable Linear Model