Estimation of ARMA processes Helle Bunzel ISU

February 17, 2009

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

1 / 55

The Box-Jenkins Principle

Historically (in the 1960’s) econometric models with many variables and equations were developed. These models …t the data beautifully. They are not good at forecasting.

It turns out that low-dimensional ARMA(p,q) processes typically do a better job with forecasting.

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

2 / 55

The Box-Jenkins Principle

The Box-Jenkins Principle for forecasting: 1

Transform the data, if nessesary, so covariance stationarity is a resonable assumption.

2

Make an initial guess at small values of p and q for an ARMA(p, q ) model that might describe the (transformed) series.

3

Estimate the parameters.

4

Perform diagnostic analysis to con…rm that the model is consistent with the observed features of the data. For Step 2 we need information on the correlation structure.

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

3 / 55

The Autocorrelation Function Recall that the autocorrelation function is de…ned as: γs ρs = p V (yt +s ) V (yt )

This is what these look like:

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

4 / 55

The Autocorrelation Function More examples

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

5 / 55

The Partial Autocorrelation Function I Clearly it is the case that it is hard to tell the processes apart from the autocorrelation function alone. Another important tool is the partial autocorrelation function. De…nition: First de-mean y , such that yt = yt µ Then the …rst partial autocorrelation is φ11 such that yt = φ11 yt

1

+ ut

The second partial autocorrelation is φ22 where yt = φ21 yt

1

+ φ22 yt

2

+ ut

Clearly for an AR (1) , φ22 = 0. These can be found as functions of the autocorrelations: Note that the m0 th autocorrelation simply is the last coe¢ cient in the linear regression of yt µ on it’s own m most recent values. Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

6 / 55

The Partial Autocorrelation Function II

Recall that we can write the coe¢ cients of the projection of yt +1 on Xt as 1 α0 = E yt +1 Xt0 E Xt0 Xt Here Xt0 = [yt

1

Xt0

1 , yt 2 , ..., yt m ]

= [yt

µ, yt

2

µ, ..., yt

m

µ] ,

And we’re projecting yt , not yt +1.

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

7 / 55

The Partial Autocorrelation Function III Now note that

E Xt0 Xt

and

2

E yt 1 yt 1 E yt 1 yt 2 6 E y y E yt 2 yt 2 t 2 t 1 = 6 4 : : E yt m yt 1 E yt m yt 2 2 3 γ0 γ1 ... γm 1 6 γ1 γ0 . γm 2 7 7 = 6 4 : : . : 5 γm 1 γm 2 ... γ0

E yt Xt0 =

Helle Bunzel (ISU)

E yt yt

1

E yt yt

Estimation of ARMA processes

2

... E yt 1 yt . E yt 2 yt . : ... E (yt m yt

... E (yt yt

3

m m m)

7 7 5

m)

February 17, 2009

8 / 55

The Partial Autocorrelation Function IV This means that we could …nd the partial autocorrelations by taking the m0 th entry of α where

α0 =

γ1 γ2 ... γm

2 6 6 4

Note that we need to invert an m

Helle Bunzel (ISU)

γ0 γ1 : γm

γ1 γ0 : 1

γm

2

3 ... γm 1 . γm 2 7 7 . : 5 ... γ0

1

m matrix to get this one.....

Estimation of ARMA processes

February 17, 2009

9 / 55

The Partial Autocorrelation Function V The partial autocorrelation of di¤erent models look like:

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

10 / 55

The Partial Autocorrelation Function VI

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

11 / 55

The Autocorrelation Functions An overview of the theoretical properties of the correlation functions:

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

12 / 55

Sample Autocorrelations I We want to calculate the autocorrelations from the data and see how they match up to the theoretical autocorrelations of various processes. This will help us …nd the correct model. We use the following as sample autocorrelations. These are basically estimates of the parameters. µˆ = y¯ =

σˆ 2 = rs =

Helle Bunzel (ISU)

1 T

∑Tt=s +1

1 T

T

∑ yt

t =1

T

∑ ( yt

y¯ )2

t =1

(yt

∑Tt=1

y¯ ) (yt

(yt

y¯ )

Estimation of ARMA processes

s

y¯ )

2

February 17, 2009

13 / 55

Sample Autocorrelations II Just as with any other estimator we need to know the variance of these estimates. If fyt g is stationary with normally distributed errors, Box and Jenkins found that ( 1 if s = 1 T V (rs ) = s 1 2 1 if s > 1 T 1 + 2 ∑j =1 rj How to use this? For an MA (s ) process, γs is 0. For large samples, rs , is normally distributed. This is enough information to create a con…dence interval.

Use the con…dence interval to test: First look at r1 . If r1 falls outside the con…dence interval, we reject γ1 = 0. Then we can look at r2 etc. Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

14 / 55

Sample Autocorrelations III

Do enough of these and one will fall outside the con…dence interval. (size!!) Ljung-Box Statistic: s

Q = T (T + 2)

∑

k =1

rk2 T

k

If all the autocorrelations are zero this is χ2 (s ) . Can also be used to check that the residual are white noise. Then under the null Q χ2 (s Q χ2 (s p q 1)

Helle Bunzel (ISU)

p

q ) , or, if a constant is included,

Estimation of ARMA processes

February 17, 2009

15 / 55

Goodness of Fit Just as in standard models you can asses goodness of …t. Fit can always be improved by adding to p and q. Criteria which trade of …t vs a more parsemonious model: Akaike information critereon: AIC = T ln (SSR ) + 2n Schwartz Bayesian criterion: SBC = T ln (SSR ) + n ln T

Where n : number of estimated parameters and T : number of usable observations. Keep T the same even if estimating …rst an AR (1) and then an AR (2) SBC punishes additional parameters harder. SBC is consistent, AIC is not. Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

16 / 55

Estimation of ARMA models. I A data example

In general, estimation of AR models can be done with simple regressions, but MA processes are more complicated. For now let the computer do the estimation. We will concern ourselves with selecting the right model. Suppose 100 data points are generated according to: yt = 0.7yt

1

+ εt

First calculate the sample (partial) auto correlations. The …rst three sample autocorrelations are r1 = 0.74, r2 = 0.58 and r3 = 0.47. Note that γ1 = 0.7, γ2 = 0.49 and γ3 = 0.343. The …rst sample partial autocorrelation is 0.71. Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

17 / 55

Estimation of ARMA models. II A data example

Graph of sample correlations:

Recall that V (rs ) =

(

1 T 1 T

1 + 2 ∑sj =11 rj2

if s = 1 if s > 1

And that if we have an MA (q ) then ρq +1 = 0. Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

18 / 55

Estimation of ARMA models. III A data example

Test MA (0) vs MA (1) . If the process is an MA (0) , then the standard deviation of r1 is 0.1. We had r1 = 0.74. Clearly reject that it is 0. V (r2 ) = 0.021 and the standard deviation is 0.1449. Again we reject that r2 is 0. Seems like we might have some action.

For the partial autocorrelation only lags 1 and 12 seem to be signi…cant. Either AR (1), AR (12) or ARMA(1, 1). Box Jenkins would indicate looking only at AR (1) and ARMA(1, 1) unless we have monthly data:

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

19 / 55

Estimation of ARMA models. IV A data example

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

20 / 55

Estimation of ARMA models. For further diagnostics, we can plot the sample autocorrelations of the residuals:

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

21 / 55

Estimation of ARMA models. I More on the example: Does there seem to be a model which allows most of PACF to be 0, but not at lag 12? First consider the ACF and the PACF for the ARMA(1,1) model: yt = a1 yt

1

+ εt + b1 εt

1

First ACF. From Yule-Walker you get:

(yt εt ) + b1 E (yt εt = a1 γ1 + σ + b1 E (yt εt 1 ) 1) + E

E (yt yt ) = a1 E (yt yt γ0

E (yt εt

Helle Bunzel (ISU)

1)

2

1)

= E ((a1 yt 1 + εt + b1 εt 1 ) εt = a1 σ2 + b1 σ2 = (a1 + b1 ) σ2 Estimation of ARMA processes

,

1)

February 17, 2009

22 / 55

Estimation of ARMA models. II γ0 = a1 γ1 + σ2 + b1 (a1 + b1 ) σ2 = a1 γ1 + [1 + b1 (a1 + b1 )] σ2 E (yt

1 yt )

γ1

= a1 E (yt 1 yt 1 ) + E (yt = a1 γ0 + b1 σ2

1 εt ) + b1 E

(yt

1 εt 1 )

,

Solve: γ0 = a1 γ1 + [1 + b1 (a1 + b1 )] σ2

γ0

= a12 γ0 + a1 b1 σ2 + [1 + b1 (a1 + b1 )] σ2 , 1 + 2a1 b1 + b12 2 = σ 1 a12 a1 + a12 b1 + a1 b12 + b1 2 σ , 1 a12 (a1 + b1 ) (1 + a1 b1 ) 2 σ 1 a12

γ1 = a1 γ0 + b1 σ2 = γ1 = Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

23 / 55

Estimation of ARMA models. III E ( yt

2 yt )

γ2

= a1 E (yt = a1 γ 1

2 yt 1 ) + E

(yt

2 εt ) + b1 E

(yt

2 εt 1 )

,

So, declining ACF..... Now PACF, Recall we need the m0 th entry of α :

α0 =

γ1 γ2 ... γm

φ11 =

Helle Bunzel (ISU)

2 6 6 4

γ0 γ1 : γm

γ1 γ0 : 1

γm

2

3 ... γm 1 . γm 2 7 7 . : 5 ... γ0

1

γ1 (a1 + b1 ) (1 + a1 b1 ) = γ0 1 + 2a1 b1 + b12

Estimation of ARMA processes

February 17, 2009

24 / 55

Estimation of ARMA models. IV α0 =

=

φ22 =

=

h

1

γ0 γ1 γ1 γ0

γ1 γ2 γ1 2 0 γ1

γ 0 γ2

γ2 2 0 γ1

γ0 γ2 γ21 = γ20 γ21

γ2 γ0

1

γ21 γ21

γ2 2 0 γ1

γ 1 γ2

γ 0 γ2

γ1 γ0 γ1 γ0

2 2

γ20

γ

=

a1 γ 1 0

1

i 2

γ1 γ0 γ1 γ0

2

a1 φ11 φ211 1 φ211 φ22 = 0 , φ11 = a1

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

25 / 55

Estimation of ARMA models. V

α0 =

γ1 γ2 γ3

2

3 γ0 γ1 γ2 4 γ1 γ0 γ1 5 γ2 γ1 γ0

1

γ21 γ0 γ2 γ2 γ1 2 2 2 γ0 + γ2 γ0 2γ21 (γ0 γ2 ) (γ0 2γ1 + γ0 γ2 ) γ20 γ21 γ3 + γ1 γ0 γ1 γ1 γ2 γ20 + γ2 γ0 2γ21 # " γ3 γ20 γ21 γ1 γ21 γ0 γ2 = γ2 + γ0 γ1 γ1 γ2 (γ20 2γ21 + γ0 γ2 ) (γ0 γ2 )

φ33 = γ1

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

26 / 55

Estimation of ARMA models. VI

γ γ2 γ21 γ21 γ0 γ2 γ2 + 3 0 γ0 γ1 γ1 γ2 ( γ0 γ2 ) 3 2 2 γ1 + γ1 γ2 + γ0 γ3 γ21 γ3 2γ0 γ1 γ2 = γ1 ( γ0 γ2 ) 3 3 2 φ + a1 φ11 + a12 φ11 a12 φ311 2a1 φ211 = γ0 11 φ11 (1 a1 φ11 )

= γ0

φ311 + a12 φ11 2a1 φ211 φ2 + a12 2a1 φ11 = γ0 11 φ11 (1 a1 φ11 ) (1 a1 φ11 )

(φ11 a1 )2 = γ0 (1 a1 φ11 )

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

27 / 55

Estimation of ARMA models. VII

φ33 =

=

"

γ21 γ0 γ2 ( γ0 γ2 )

(γ20

γ1 2γ21 + γ0 γ2 )

(γ20

(φ11 a1 )2 γ1 γ0 2γ21 + γ0 γ2 ) (1 a1 φ11 )

γ γ2 γ21 γ2 + 3 0 γ0 γ1 γ1 γ2

#

So, for φ33 = 0 and φ11 > 0, we’d need φ11 = a1 Why is this not compatible with our model?

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

28 / 55

Estimation of ARMA models. VIII You can also see this by solving: φ11 =

(a1 + b1 ) (1 + a1 b1 ) = a1 1 + 2a1 b1 + b12

(a1 + b1 ) (1 + a1 b1 ) 1 + 2a1 b1 + b12 (a1 + b1 ) (1 + a1 b1 ) a1 1 + 2a1 b1 + b12 b1 (a1 1) (a1 + 1) b1

Helle Bunzel (ISU)

Estimation of ARMA processes

= a1 , = 0, = 0, = 0

February 17, 2009

29 / 55

Estimation of ARMA models. IX Now calculate the ACF and the PACF for the model yt = a1 yt

1

+ εt + b12 εt

12

From Yule-Walker you get: E (yt yt ) = a1 E (yt yt

E (yt εt

γ0

(yt εt ) + b12 E (yt εt = a1 γ1 + σ + b12 E (yt εt 12 )

12 )

= E = E = =

Helle Bunzel (ISU)

1) + E

12 )

2

"

1 (εt + b12 εt 1 a1 L ∞

∑

a1i εt i

i =0 12 2 E a1 εt 12 a112 + b12

12 )

∞

+ b12

∑

i =0

+ b12 ε2t σ2

εt

,

12

a1i εt 12 i

!#

εt

12

!

12

Estimation of ARMA processes

February 17, 2009

30 / 55

Estimation of ARMA models. X γ0 = a1 γ1 + γ0 + b12 E (yt εt γ 0 = a1 γ 1 + σ E (yt

1 yt )

γ1

E (yt

2

+ b12 a112

12 )

+ b12

, σ2

= a1 E (yt 1 yt 1 ) + E (yt 1 εt ) + b12 E (yt = a1 γ0 + b12 E (yt 1 εt 12 )

1 εt 12 )

= E = E

"

1 ( εt 1 a1 L

1

+ b12 εt

∞

∑ a1i εt

13 )

εt

1 εt 12 )

12

∞

1 i

+ b12

i =0

∑ a1i εt

i =0

,

13 i

!#

εt

12

!

= a111 σ2 γ1 = a1 γ0 + b12 a111 σ2 Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

31 / 55

Estimation of ARMA models. XI Plug into …rst equation γ0 = a1 γ1 + σ2 + b12 a112 + b12 σ2 ,

γ0 = a1 a1 γ0 + b12 a111 σ2 + σ2 + b12 a112 + b12 σ2 1

a12 γ0 = b12 a112 σ2 + σ2 + b12 a112 + b12 σ2 γ0 =

2 1 + 2b12 a112 + b12 σ2 1 a12

γ1 = a1 γ0 + b12 a111 σ2 2 1 + 2b12 a112 + b12 σ2 + b12 a111 σ2 = a1 1 a12

Helle Bunzel (ISU)

=

2 a + b a11 a1 + 2b12 a113 + b12 1 12 1 2 1 a1

=

2 a + b a11 a1 + b12 a113 + b12 1 12 1 σ2 2 1 a1 Estimation of ARMA processes

b12 a113

σ2

February 17, 2009

32 / 55

Estimation of ARMA models. XII

E (yt

2 yt )

γ2

E (yt

= a1 E (yt 2 yt 1 ) + E (yt 2 εt ) + b12 E (yt = a1 γ1 + b12 E (yt 2 εt 12 )

2 εt 12 )

= E = E

"

1 ( εt 1 a1 L ∞

∑

2

a1i εt 2 i

+ b12 εt

14 )

∞

+ b12

i =0

∑

i =0

εt

2 εt 12 )

,

12

a1i εt 14 i

!#

εt

12

!

= a110 σ2

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

33 / 55

Estimation of ARMA models. XIII γ2 = a1 γ1 + b12 a110 σ2 2 a + b a11 a1 + b12 a113 + b12 1 12 1 = a1 σ2 + b12 a110 σ2 1 a12

=

2 a2 + b a12 + b a10 a12 + b12 a114 + b12 12 1 12 1 1 1 a12

=

2 a2 + b a10 a12 + b12 a114 + b12 12 1 1 σ2 1 a12

2 + β a12 1 + β12 a112 + b12 12 1 γj = 1 a12

γ j = a1 γ j

Helle Bunzel (ISU)

1,

j

Estimation of ARMA processes

b12 a112

σ2

2j

a1j σ2 , 1

j

12

13

February 17, 2009

34 / 55

Estimation of ARMA models. XIV This provides autocorrelation coe¢ cients: γ ρ1 = 1 = γ0 When 2

j

2 a +b a 11 a 1 +b 12 a 113 +b 12 1 12 1 σ2 1 a 12 2 1 +2b 12 a 112 +b 12 2 1 a1

2 1 + b12 a112 + b12 a110 + b12 2 1 + 2b12 a112 + b12

12

ρj

=

= and ρj = Helle Bunzel (ISU)

σ2

= a1

γj = γ0

2 +b a 12 1 +b 12 a 112 +b 12 12 1 1 a 12

2j

a1j σ2

2 1 +2b 12 a 112 +b 12 σ2 1 a 12 2 + b a12 2j 1 + b12 a112 + b12 12 1 a1j 2 1 + 2b12 a112 + b12

γj a1 γ j = γ0 γ0

1

= a1 ρ j

Estimation of ARMA processes

1,

j

13 February 17, 2009

35 / 55

Estimation of ARMA models. XV So, decaying autocorrelations after lag 12. Now, the PACF. Starting with the second: α0 = h

= Thus,

φ22 = φ33 =

(γ20

γ1 γ2

γ0 γ1 γ1 γ0

γ0 γ1 γ1 γ2 γ20 γ21

γ0 γ2 γ21 γ20 γ21

1

i

ρ2 ρ21 γ0 γ2 γ21 = γ20 γ21 1 ρ21 " # (φ11 a1 )2 γ1 2γ21 + γ0 γ2 ) (1 a1 φ11 )

For this to be 0 we need ρ2 = ρ21 Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

36 / 55

Estimation of ARMA models. XVI but that implies 2 +b a 8 1 +b 12 a 112 +b 12 12 1 2 a1 2 1 +2b 12 a 112 +b 12

= a12

2 1 +b 12 a 112 +b 12 a 110 +b 12 2 1 +2b 12 a 112 +b 12

2 1 + b12 a112 + b12 + b12 a18

=

2 1 + b12 a112 + b12 a110 + b12

β12 a18 (a1

2

,

2 1 + 2b12 a112 + b12 2

2 1)2 (a1 + 1)2 b12 + b12 a112 + 1 = 0 ,

b12 = 0 _ a1 = 0

Not 0 for this model. Clearly PACF 2 Helle Bunzel (ISU)

11 are NOT 0 for this model. Estimation of ARMA processes

February 17, 2009

37 / 55

Estimation of ARMA models. XVII Another model candidate would be yt = a1 yt

1

+ a2 yt

12

+ εt

Calculate ACF: E (yt yt ) = a1 E (yt yt γ0 = E (yt

1 yt )

γ1 E (yt

2 yt )

γ2 E (yt

3 yt )

γ3 Helle Bunzel (ISU)

1 ) + a2 E a1 γ1 + a2 γ12 + σ2

(yt yt

12 ) + E

( yt ε t ) ,

= a1 E (yt 1 yt 1 ) + a2 E (yt = a1 γ0 + a2 γ11

1 yt 12 ) + E

(yt

1 εt )

,

= a1 E (yt 2 yt 1 ) + a2 E (yt = a1 γ1 + a2 γ10

2 yt 12 ) + E

(yt

2 εt )

,

= a1 E (yt 3 yt 1 ) + a2 E (yt = a1 γ 2 + a2 γ 9

3 yt 12 ) + E

(yt

3 εt )

,

Estimation of ARMA processes

February 17, 2009

38 / 55

Estimation of ARMA models. XVIII

Skip to the 12th: E (yt

12 yt )

γ12 E (yt

13 yt )

γ13

= a1 E (yt 12 yt 1 ) + a2 E (yt = a1 γ11 + a2 γ0

12 yt 12 ) + E

(yt

12 εt )

,

= a1 E (yt 13 yt 1 ) + a2 E (yt = a1 γ12 + a2 γ1

13 yt 12 ) + E

(yt

13 εt )

,

We have 13 equations with 13 unknows.

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

39 / 55

Estimation of ARMA models. XIX Computer or much patience can solve these. γ0 γ1 γ2 γ3 γ4 γ5 γ6 γ7 γ8 γ9 γ10 γ11 γ12 Helle Bunzel (ISU)

= = = = = = = = = = = = =

a1 γ1 + a2 γ12 + σ2 a1 γ0 + a2 γ11 a1 γ1 + a2 γ10 a1 γ 2 + a2 γ 9 a1 γ 3 + a2 γ 8 a1 γ 4 + a2 γ 7 a1 γ 5 + a2 γ 6 a1 γ 6 + a2 γ 5 a1 γ 7 + a2 γ 4 a1 γ 8 + a2 γ 3 a1 γ 9 + a2 γ 2 a1 γ10 + a2 γ1 a1 γ11 + a2 γ0

Estimation of ARMA processes

February 17, 2009

40 / 55

Estimation of ARMA models. XX

φ11 =

φ22 =

γ1 γ0

γ0 γ2 γ21 =0, γ20 γ21

γ0 γ2 = γ21 , σ4 a110 a2 =0 K

Again this one cannot be 0 and be compatible with the model in question.....

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

41 / 55

AR(2) example

Now consider data generated by yt = 0.7yt

1

0.49yt

2

+ εt

The estimation results are:

Model looks good.

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

42 / 55

AR(2) example

ACF and PACF for the series Here there are a few that are too large. ACF 16 and PACF 17. This might have tempted us to estimate a di¤erent model. ACF for residuals is large at 14 and 17 Also Ljung-Box(16) is signi…cant. Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

43 / 55

AR(2) example Try the following model: yt = a1 yt

1

+ a2 yt

2

+ εt + β16 εt

16

Estimation results are:

This looks like a better model! Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

44 / 55

Parameter Estimation I Estimation of AR models

The simple-minded approach would be to run regression of y on p lags of y and use OLS estimates Let’s consider the properties of this estimator Use AR (1) as example Assume y0 is available

Then: ρˆ =

∑Tt=1 (yt ∑Tt=1 (yt 1

= ρ+

1

y¯t

1 ) (yt

y¯t ) y¯t 1 )

y¯t 1 ) (yt 1 T ∑t =1 (yt 1 y¯t 1 ) (εt T ∑t =1 (yt 1 y¯t 1 ) (yt 1

ε¯ t ) y¯t 1 )

We would like to answer questions about bias, consistency, variance etc Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

45 / 55

Parameter Estimation II Estimation of AR models

We have to regard ‘regressor’as stochastic. It is not clear that ( E

∑Tt=1 (yt ∑Tt=1 (yt 1

y¯t

1

y¯t

1 ) ( εt

1 ) (yt 1

ε¯ t ) y¯t 1 )

)

=0

We can’t derive explicit expression for bias, but it can be shown that: The OLS estimate is biased and bias is negative. This bias often called Hurwicz bias – it can be sizeable in small samples Hurwicz bias goes to zero as T ! ∞ The OLS estimate is consistent The OLS estimator is asymptotically normal with usual formulae for asymptotic variance.

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

46 / 55

Parameter Estimation III Estimation of AR models

Note that to estimate an AR (p ) model by OLS does not use information contained in …rst p observations This causes a loss of e¢ ciency from this

There are a number of methods which use this information.

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

47 / 55

Maximum Likelihood Parameter Estimation, AR processes I

Now consider estimation of an AR (1) process using maximum likelihood. Assume the errors are normal white noise. The model is: yt = c + φyt

1

+ εt

We wish to estimate c, φ and σ2 . The …rst observation y1 is normally distributed with E (y1 ) = µ = V (y1 ) =

Helle Bunzel (ISU)

c 1

φ

σ2 1 φ2

Estimation of ARMA processes

February 17, 2009

48 / 55

Maximum Likelihood Parameter Estimation, AR processes II The density of this observation is

fY 1 y1 ; c, φ, σ2 = q

1 2 2π 1 σ φ2

Now consider y2 . We know that

0

B exp @

y1

c 1 φ 2 2 1 σ φ2

2

1 C A

y2 = c + φy1 + ε2 We can easily see that fY 2 jY 1 y2 jy1 ; c, φ, σ Helle Bunzel (ISU)

2

=p

1 2πσ2

exp

Estimation of ARMA processes

(y2

(c + φy1 ))2 2σ2 February 17, 2009

! 49 / 55

Maximum Likelihood Parameter Estimation, AR processes III Recall that fY 1 ,Y 2 = fY 2 jY 1 fY 1 Similarly y3 = c + φy2 + ε3 and fY 3 jY 1 ,Y 2 y3 jy1 , y2 ; c, φ, σ

2

=p

1 2πσ2

(y3

exp

(c + φy2 ))2 2σ2

!

Continuing, we get fY 1 ,Y 2 ,...,Y T y1 , y2 , ..., yT ; c, φ, σ2

= fY1 y1 ; c, φ, σ2

T

∏ fY j Y t

t 1

t =2 Helle Bunzel (ISU)

Estimation of ARMA processes

yt jyt

1 ; c, φ, σ

2

February 17, 2009

50 / 55

Maximum Likelihood Parameter Estimation, AR processes IV Then the log-likelihood is

L c, φ, σ2

=

1 log (2π ) 2 T

1 2

T

∑

t =2

1 log 2

log (2π )

(yt

y1

σ2 φ2

1 T

1

2 (c + φyt 1 ))2 2σ2

c

2

1 φ 2 2 1 σ φ2

log σ2

This one must be maximized numerically. Another option is to do maximum likelihood conditional on the …rst observation. This turns out to give the same result as the OLS estimation with the same drawbacks. Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

51 / 55

Maximum Likelihood Parameter Estimation, MA processes I We will consider a MA (1) process: yt = µ + εt + θεt

1

Assume the errors are normal white noise. We wish to estimate µ, θ and σ2 . If we knew the value of εt fY t j ε t

1

yt j ε t

1 ; µ, θ, σ

2

1,

we could write

=p

1 2πσ2

exp

(yt

µ θεt 2σ2

1)

2

!

Now, if we knew with certainty that ε0 = 0, y1 = µ + ε1 Helle Bunzel (ISU)

N µ, σ2

Estimation of ARMA processes

February 17, 2009

52 / 55

Maximum Likelihood Parameter Estimation, MA processes II Note that this trick is similar to assuming the …rst observation was known for the AR (1) process. Also, when y1 is know, so is ε1 = y1

µ. Therefore

fY 2 jY 1 ,ε0 =0 y2 jy1 , ε0 = 0; µ, θ, σ2 = p

1 2πσ2

exp

(y2

µ θε1 )2 2σ2

!

The, since y2 = µ + ε2 + θε1 ε2 is also known when ε1 is.

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

53 / 55

Maximum Likelihood Parameter Estimation, MA processes III Continue like this to …nd the density functions. fY t j Y t

= fY t j ε t =

p

1 ,Y t 2 ,...,Y 1 ,ε0 =0 1

1

2πσ2

yt jεt exp

yt jyt

1 ; µ, θ, σ

1 , yt 2 , ..., y1 , ε0

= 0; µ, θ, σ2

2

ε2t 2σ2

From this we get the log-likelihood function

L µ, θ, σ2 =

T log (2π ) 2

T log σ2 2

T

ε2

∑ 2σt 2

t =2

where εt is calculated recursively from the data. It is not simple to get the MLE from this expression, so numerical methods must be used to maximize the likelihood function. Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

54 / 55

Maximum Likelihood Parameter Estimation, MA processes IV

It turns out that if jθ j < 1, the e¤ect of assuming ε0 = 0 dies out as T becomes big. There are also many methods for calculating the MLE for MA processes without this assumption, but these too are non-trivial numerical methods.

Helle Bunzel (ISU)

Estimation of ARMA processes

February 17, 2009

55 / 55