## The Multiple Regression Model and theleast Squares Assumptions

The Multiple Regression Model and theLeast Squares Assumptions Yi = β0 + β1X1i + β2X2i + … + βkXki + ui, i = 1,…,n 1. The conditional distribution of...
Author: Shana Barnett
The Multiple Regression Model and theLeast Squares Assumptions Yi = β0 + β1X1i + β2X2i + … + βkXki + ui, i = 1,…,n

1. The conditional distribution of u given the X’s has mean zero, that is, E(u|X1 = x1,…, Xk = xk) = 0. 2. (X1i,…,Xki,Yi), i =1,…,n, are i.i.d. 3. X1,…, Xk, and u have four moments: E( X 1i4 ) < ∞,…, E( X ki4 ) < ∞, E( ui4 ) < ∞. 4. There is no perfect multicollinearity.

The OLS estimator of the β’s is unbiased and consistent. For large n,

βˆi − β i se( βˆi )

is approximately

distributed N(0,1)

Consider the following example using the CA test score data set: TESTSCRi = β0 + β1STRi + β2EXPN_STUi + β3AVG_INCi + β4EL_PCTi + ui TESTSCR = average test score STR = student-teacher ratio EXPN_STU = expenditures per student (\$) AVG_INC = average household income (\$) EL_PCT = percent of English learners

Estimate this model by OLS in Eviews, with heteroskedasticity adjusted standard errors to obtain:

Dependent Variable: TESTSCR Method: Least Squares Date: 09/30/04 Time: 14:35 Sample: 1 420 Included observations: 420 White Heteroskedasticity-Consistent Standard Errors & Covariance Variable

Coefficient

Std. Error

t-Statistic

Prob.

C STR EXPN_STU AVGINC EL_PCT

651.7362 -0.323907 -0.001289 1.518165 -0.483630

11.42310 0.360775 0.001098 0.092780 0.028504

57.05423 -0.897807 -1.173235 16.36306 -16.96727

0.0000 0.3698 0.2414 0.0000 0.0000

R-squared Adjusted R-squared S.E. of regression Sum squared resid Log likelihood Durbin-Watson stat

0.708237 0.705425 10.34116 44379.90 -1574.614 1.193905

Mean dependent var S.D. dependent var Akaike info criterion Schwarz criterion F-statistic Prob(F-statistic)

654.1565 19.05335 7.521974 7.570072 251.8473 0.000000

First, let’s look at the point estimates of the slope coefficients. • The signs of the estimated coefficients seem to conform to our expectations, except for the negative coefficient on EXPN_STU. • The coefficient on STR is much smaller (in absolute value) than in the other regressions we have looked at.

Hypothesis Testing: Under the null hypothesis H0:βi =βi,0 for large n, the t-statistic t=

βˆi − β i 0 se( βˆi )

is approximately N(0,1). So tests of this null hypothesis can be applied in exactly same way as in the simple regression model.

Example: California Test Scores First, consider the null hypothesis that β1, the coefficient on STR, is equal to 0 (i.e., all else equal, variations in STR do not affect test scores) against the one-sided alternative that the coefficient is less than zero. The t-statistic corresponding to this null hypothesis is -0.898 (= -.324/.361) The p-value for this one-sided test is Prob( t < -.898), which is the probability that an N(0,1) random variable is less than -.898 which is approximately 0.19. Therefore, we do not reject the null hypothesis at conventional significance levels (e.g., 10% ,5%, 1%).

The coefficient on EXPN_STU appears to have the “wrong” sign, suggesting that increasing expenditures per student will, all else equal, reduce test scores. Let’s test the null hypothesis that β2, the coefficient on EXPN_STU is equal to 0 against the two-sided alternative that it is not equal to 0. The t-statistic is equal to -1.17. The pvalue for this two-sided test is equal to the probability of drawing a N(0,1) random variable that is larger than 1.17 in absolute value, which is equal to 0.24. Therefore, we would not reject the null hypothesis that β2 = 0 against the two-sided alternative at conventional significance levels.

It appears that, all else equal (including expenditures per student), changes in STR do not affect test scores. It also appears that all else equal (including studentteacher ratio) that changes in EXPN_STU do not affect test scores. Does it follow that, all else equal, neither changes in STR nor changes EXPN_STU affect test scores? No. If STR and EXPN_STU are correlated with one another, the individual t-tests could easily be “misled” into understating the actual significance of each of these variables. The appropriate way to test the hypothesis that neither one matters is to formulate a joint hypothesis and apply an F-test.

The joint null hypothesis that β1 = 0 and β2 = 0 is: H0: β1 = 0,β2 = 0 The alternative hypothesis is that at least one of (β1,β2) is non-zero. The F-statistic can be constructed to test this joint hypothesis. Under the null hypothesis, for large n, the F-statistic will have an F(2,∞) distribution. (Or equivalently, 2×F will have a chi-square distribution with 2 degrees of freedom.)

The null hypothesis is rejected at the αsignificance level if the statistic is greater than the (1-α)x100 percentile of the F(2, ∞) distribution. The p-value of the test is the probability of drawing a number from the F(2, ∞) distribution larger than the calculated value of the statistic.

Note that the “correct” calculation of the F-statistic will depend on whether you assume that the regression errors are heteroskedastic or whether you make the more restrictive assumption that they are homoskedastic.

To calculate the F-statistic in Eviews: Estimate the original regression equation by OLS, using the heteroskedasticity-consistent standard errors option (unless we assume that the errors are homoskedastic) Apply the Wald test with the restrictions “C(2)=0,C(3)=0” (assuming that STR and EXPN_STU are the second and third variables in the list under “variables” in the regression)

Output:

Wald Test: Equation: Untitled Null C(2)=0 Hypothesis: C(3)=0 F-statistic 0.720996 Probability Chi-square 1.441992 Probability

0.486876 0.486268

This produces F and 2xF (the Chi-square statistic). The “Probability” is the pvalue of the test. That is under the null hypothesis that C(2)=0 and C(3)=0, the probability of drawing an F-statistic as large as 0.72 is about 0.49. (Why are the two probabilities different?) So, it appears that neither STR nor EXPN_STU have statistically significant effects on test scores.

Maybe none of the variables that we’ve included affects test scores! Let’s test this joint null hypothesis – H0 : β1 = 0, β2 = 0, β3 = 0, β4 = 0 against the two-sided alternative that at least one of these β’s is equal to zero. Under the null hypothesis, for large n, the F-statistic has an F(4,∞) distribution (and, equivalently, 4xF has a chi-squared distribution with 4 degrees of freedom). We reject H0 at the α-significance level if the calculated F is greater than the (1-α)x100 percentile of the F(4,∞) distribution.

In Eviews, we follow the same procedure as we used for the previous F-test, except that now we insert the following restrictions into the Wald test dialogue box: C(2)=0,C(3)=0,C(4)=0,C(5)=0

The result of this test: Wald Test: Equation: Untitled Null Hypothesis: C(2)=0 C(3)=0 C(4)=0 C(5)=0 F-statistic Chi-square

223.0756 892.3023

Probability Probability

0.000000 0.000000

Note that the Chi-square statistic is 4 times the F-statistic. Note, too that the F-statistic different from the F-statistic produced in the regression summary, even though these F-statistics have been constructed to test the same hypothesis (i.e., all of the slope coefficients are equal to 0). Why?

The p-value of this test is VERY small and so we reject the joint null hypothesis at standard significance levels. So, something seems to matter! What seem to matter are the district characteristic that policymakers have little control over – average income and English learners in the district. Each of these two variables is statistically significant.

The joint hypothesis we have looked at so far have been use of the form that some subset of the regression parameters are equal to zero. However, the F-test can be applied to test more general kinds of linear restrictions. Example 1: H0: β1 = β3 The F-statistic will have an F(1,∞) distribution, for large n. Eviews implementation? Same as above, but in the Wald-test dialogue box use the restriction C(2)=C(4), assuming the intercept appears at the top of the variable list.

Example 2: H0: β1 = β3 and 2β1 + 0.5β2 = β4 The F-statistic will have an F(2,∞) distribution, for large n. Eviews implementation? Same as above, but in the Wald-test dialogue box use the restriction C(2)=C(4), 2*C(2)+0.5*C(3)=C(5) again assuming the intercept appears at the top of the variable list.

More on the F-statistic – The F-statistic essentially compares how well the estimated model fits the data when the restrictions specified by the null hypothesis are imposed against the fit of the model when the restrictions are ignored. If the restrictions are correct, then imposing the restrictions should not substantially worsen the fit of the model. This is especially clear in the case where the errors are assumed to be homoskedastic. In this special case, the F-statistic has the following form:

Run two regressions, one under the null hypothesis (the “restricted” regression) and one under the alternative hypothesis (the “unrestricted” regression), the “restricted” and “unrestricted” regressions. 2 2 ( Runrestricted − Rrestricted )/q F= 2 (1 − Runrestricted ) /( n − kunrestricted − 1)

where: 2 Rrestricted = the R2 for the restricted regression 2 Runrestricted = the R2 for the unrestricted

regression q = the number of restrictions under the null kunrestricted = the number of regressors in the unrestricted regression.

Example: are the coefficients on STR and Expn zero? Restricted population regression (that is, under H0): TESTSCRi = β0 + β3AVG_INCi + β4EL_PCTi + ui (why?) Unrestricted population regression (under H1): TESTSCRi = β0 + β1STRi + β2EXPN_STUi + β3AVG_INCi + β4EL_PCTi + ui

• The number of restrictions under H0 = q = 2. • The fit will be better (R2 will be higher) in the unrestricted regression (why?) By how much must the R2 increase for the coefficients on STR and EXPN_STU to be judged statistically significant? Under the null hypothesis, for large n and homoskedastic errors, this statistic is drawn from an F(2,∞) distribution.

When the errors are heteroskedastic, this simple form of the F-statistic does not have an F(q, ∞). The “correct” version of the F-statistic under heteroskedasticity has a more complicated form, but it works according to the same principles: If the fit of the model is substantially worse when the null hypothesis is imposed then the null hypothesis is rejected.