REPUBLIC OF SOUTH AFRICA GOVERNMENT-WIDE MONITORING & IMPACT EVALUATION SEMINAR

Session IV Difference in Differences (Panel Data) June 2006

ORGANIZED BY THE WORLD BANK AFRICA IMPACT EVALUATION INITIATIVE IN COLLABORATION WITH HUMAN DEVELOPMENT NETWORK AND WORLD BANK INSTITUTE

1

Employing Panel Data • Assume one is interested in the effect of a treatment (I) as measured in a regression of the form (1) Y = α +γI + βX + e • Where the error term, e, is composed of individual, family and community unobserved fixed characteristics as well as a stochastic disturbance term, μ: (2) e = ν + η + ε + μ 2

Employing Panel Data • While randomization may insure that cov (I,e)=0, this is not assured in al cases Often one takes the advantage of panel data to estimate differences in difference results • Under the assumption that the first three terms of (2) are fixed, by taking first differences one can eliminate the correlation between them and the treatment by estimating • (3) ΔY = γΔI + βΔX + Δμ 3

But Simple Reflexive is Seldom Enough • As written, (3) assumes away any secular trend. If this is justified, the so-called reflexive approach is a comparatively inexpensive means of program evaluation. If a secular trend is assumed, a simple before and after comparison in the communities where a program is offered would not allow this trend to be identified separately from ΔI. Thus, in most cases a control group is still required to identify an overall intercept that can be used to measure this trend. 4

Difference in Differences • Thus, the use of Difference in Difference approaches • EDD(γ)=[E(YT1) –E(YC1)] – [E(YT0) –E(YC0)] • In practice this can be estimated by including a bivariate dummy variable for the treatment in a regression of changes in Y against changes in Xs when the intervention has been introduced in mid-panel • Alternatively, community, household, or individual fixed effects models can be used 5

Of Course, Dif-Dif not a Panacea • Community fixed effects or first differencing will not eliminate all of the potential bias if, instead of equation (1), we need to estimate an equation of the following form • (4) Y = α + γ1I + γ2(I * ε) + βX + e • This implies that in addition to the first-order consideration when unobserved community factors enter additively, community effects also enter multiplicatively. The bias, for example, will be positive if communities have different potential to benefit from an intervention 6

Different Time Trends Can Also Introduce a Bias • Compare the following two illustrations. The first is a standard dif-in-dif model in which the estimated ATE is unbiased (though an estimate without a panel would not be) • In contrast, in the second example, the treatment has a different pre-existing time trend, Thus this trend must be taken into account for proper estimation 7

Outcome

Average Treatment Effect on the Treated

Treatment Group

Control Group

Treatment

Time 8

Outcome

Average Treatment Effect on the Treated Estimated Average Treatment Effect on the Treated

Treatment Group

Control Group

Treatment

Time 9

Dif-in-Dif Need not be Bivariate Comparisons • One may want to compare marginal effects in treatment with marginal effect in control • Examples: changes in rates of growth or the impact of ranges of iron intake with and without concomitant treatment for worms • In a non-intervention study, Thomas (JHR 1994) compared difference in mothers’ investments in daughters and sons with fathers’ investments [Βg,m – Βb,m] - [Βg,f – Βb,f] 10

Can Combine Dif-in Dif with Random Assignment • Example: Randomized Assignment of communities in Uganda, all received Child Health Days, but only half got deworming treatment in the package • The treatment group had a statistically significant different extra weight gain of 154 g • But number of times a child visited the periodic Health Days is endogenous • Can compare coefficients of number of visits

11

Uganda Dif-in-Dif Results • In regression of weight gain against number of visits • ΒtV – ΒcV was 55g. That is, for each visit a child made, those children 1- 6 years old with Albendazole gained 55 extra grams • A similar pattern emerged comparing children in treatment who visited every 6 months with children with similar frequency in control; this difdif was twice that for those with annual visits • Source: Alderman et al. BMJ 2006 12

Example

Do Police Reduce Crime? Estimates using the Allocation of Police Forces after a Terrorist Attack

R. Di Tella and E. Schargrodsky AER (2005) 13

Events & Data •

The authors of this paper obtained all the information available to the police (with the exception of the victim’s name) about each auto theft in these neighborhoods for the nine-month period starting April 1, 1994 and ending December 31, 1994. Figure 1: Timeline of Events July

April

May

Pre

June

July 18 July 25

Terrorist Attack

Aug

Police Protection Ordered

Sept

Oct

Nov

Dec

Post

14

F i g u r e 2 - W e e k ly E v o lu tio n o f C a r T h e ft s 0 .0 8

0 .0 6

0 .0 4

0 .0 2

0

P R E -A TT A C K

P O S T -A T T A C K

-0 .0 2 1

2

3 A p r il

4

5

6

7

8

9

M ay

10

11

12

13

14

Ju n e

15

16 Ju l y

17

18

19

20

A u g u st

21

22

23

24

25

26

S e p te m b e r

27

28

29

30

O c to b er

31

32

33

34

N o v em b e r

35

36

3 De

W eek

J ew ish I nstitu tion in th e B loc k

P re a nd P ost M ea ns f or J ew ish I nstitu tion in th e B loc k

O ne B loc k f rom N e a re st Je w ish I nstitut ion

P re a nd P ost M ea ns f or O ne B loc k f ro m N e a re st Je w ish I nstitu tion

T w o B loc ks f rom N e a re st Je w ish I nstituti on

P re a nd P ost M ea ns f or T w o B loc ks f rom N e a re st Je w ish I nstitut ion

M or e th an T w o B loc ks f rom N e a re st Je w ish I nstituti on

P re a nd P ost M ea ns f or M or e th an T w o B loc ks f rom N e a re st Je w ish

15

Empirical Strategy

CarTheftit = α0 SameBlockPoliceit +α1OneBlockPoliceit + + α 2 Two Blocks

Police

it

+ M

t

+ F i + ε it

16

Table: The Effect of Police Presence on Car Theft

(A ) S P O P T P

a m e -B lo c k o lic e n e -B lo c k o lic e w o -B lo c k s o lic e

B E M E N R

lo c k F ix e d ffe c t o n th F ix e d ffe c t o f o b s e rv a tio n s 2

- 0 .0 7 7 5 2 * ( 0 .0 2 2 )

(B ) * *

(C )

- 0 .0 8 0 0 7 ** ( 0 .0 2 2 ) - 0 .0 1 3 2 5 ( 0 .0 1 3 )

*

- 0 .0 8 0 8 0 ** ( 0 .0 2 2 ) - 0 .0 1 3 9 8 ( 0 .0 1 4 ) - 0 .0 0 2 1 8 ( 0 .0 1 2 )

Y e s

Y e s

Y e s

Y e s

Y e s

Y e s

7 8 8 4 0 .1 9 8 3

7 8 8 4 0 .1 9 8 4

7 8 8 4 0 .1 9 8 4

N o te : D e p e n d e n t v a ria b le : n u m b e r o f c a r th e f p e r b lo c k . L e a st S q u a re s D u m m y V a ria b re g re s s io n s . C a r th e fts th a t o c c u rre d b e tw e e n J u ly 3 1 a re e x c lu d e d . H u b e r-W h ite s ta n d a rd p a re n th e s e s . *** S ig n ific a n t a t th e 1 % le v e l.

*

ts p e r m o n th le s (L S D V ) J u ly 1 8 a n d e rro rs a re in

17

Table: Monthly Coefficients S a m e - B lo c k P o li c e – A u g u s t S a m e - B lo c k P o li c e – S e p t e m b e r S a m e - B lo c k P o li c e – O c t o b e r S a m e - B lo c k P o li c e – N o v e m b e r S a m e - B lo c k P o li c e – D e c e m b e r

O n e - B lo c k P o li c e – A u gu st O n e - B lo c k P o li c e – S e p t e m b e r O n e - B lo c k P o li c e – O c t o b e r O n e - B lo c k P o li c e – N o v e m b e r O n e - B lo c k P o li c e – D e c e m b e r

T w o - B l o c k s P o li c e – A u g u s t T w o - B l o c k s P o li c e – S e p t e m b e r T w o - B l o c k s P o li c e – O c t o b e r T w o - B l o c k s P o li c e – N o v e m b e r T w o - B l o c k s P o li c e – D e c e m b e r

- 0 .0 7 9 1 4 * * ( 0 .0 3 7 ) - 0 .0 9 6 3 3 * * * ( 0 .0 2 6 ) - 0 .0 6 8 4 0 ( 0 .0 4 2 ) - 0 .0 7 7 2 9 * * * ( 0 .0 2 7 ) - 0 .0 8 2 8 2 * * * ( 0 .0 2 7 ) - 0 .0 5 4 0 0 * * ( 0 .0 2 3 ) - 0 .0 1 4 1 1 ( 0 .0 2 4 ) - 0 .0 2 5 7 0 ( 0 .0 2 4 ) 0 .0 1 1 6 0 ( 0 .0 2 6 ) 0 .0 1 2 2 8 ( 0 .0 2 6 ) 0 .0 1 0 0 9 ( 0 .0 2 3 ) - 0 .0 0 2 0 7 ( 0 .0 2 2 ) - 0 .0 3 1 3 8 ( 0 .0 1 9 ) 0 .0 0 6 7 7 ( 0 .0 1 9 ) 0 .0 0 5 6 6 ( 0 .0 2 0 )

18

Table: Car Thefts Before the Terrorist Attack

S P O P T P

a m e -B lo c k o lic e n e -B lo c k o lic e w o -B lo c k s o lic e

B lo c k F ix e d E ffe c t M o n th F ix e d E ffe c t N o f o b s e rv a tio n s R 2

(A ) P o lic e d u m m ie s a c tiv a te d o n A p ril 3 0

(B ) P o lic e d u m m ie s a c tiv a te d o n M ay 3 1

(C ) P o lic e d u m m ie s a c tiv a te d o n Ju n e 3 0

- 0 .0 1 8 6 4 ( 0 .0 5 3 ) - 0 .0 2 5 5 3 ( 0 .0 2 5 ) - 0 .0 3 2 6 3 ( 0 .0 2 2 )

0 .0 1 4 6 7 ( 0 .0 4 0 ) 0 .0 1 4 0 2 ( 0 .0 1 9 ) - 0 .0 1 4 6 5 ( 0 .0 1 7 )

- 0 .0 3 6 1 1 ( 0 .0 3 8 ) 0 .0 2 3 1 0 ( 0 .0 2 2 ) - 0 .0 0 9 4 0 ( 0 .0 1 6 )

Y es

Y es

Y es

Y es

Y es

Y es

3 5 0 4

3 5 0 4

3 5 0 4

0 .3 2 0 6

0 .3 2 0 2

0 .3 2 0 4

N o te : D e p e n d e n t v a ria b le : n u m b e r o f c a r th e fts p e r m o n th p e r b lo c k . L e a s t S q u a re s D u m m y V a ria b le s (L S D V ) re g re s s io n s . S a m p le p e rio d : A p ril 1 J u ly 1 7 . T h e v a ria b le S a m e -B lo c k P o lic e in c o lu m n (A ) e q u a ls 1 b e tw e e n A p ril 3 0 a n d J u ly 1 7 (fo r b lo c k s th a t c o n ta in a J e w is h in s titu tio n ) a n d 0 o th e rw is e . T h e s a m e is tru e fo r O n e -B lo c k P o lic e , a n d T w o -B lo c k s P o lic e (fo r b lo c k s o n e b lo c k a w a y fro m th e n e a re s t J e w is h in s titu tio n a n d b lo c k s tw o b lo c k s a w a y fro m th e n e a re s t J e w is h in s titu tio n , re s p e c tiv e ly ). C o lu m n (B ) re d e fin e s th e s e v a ria b le s u s in g M a y 3 1 , a n d c o lu m n (C ) u s e s J u n e 3 0 . H u b e r-W h ite s ta n d a rd e rro rs a re in p a re n th e s

19

Policy Discussion •

•

•

How much crime is reduced if we hire an extra policeman? The results of this paper suggest that the effect is large, but extremely local. Police presence reduces car thefts in the block, and no effect just one block away is found. During this period, a Buenos Aires policeman was earning on average a monthly wage of $800. Given that policemen work eighthour shifts and average 21 work days per month, the monthly cost of providing police protection for one block is approximately $3,428. The estimates of this paper suggests that police presence in a block would induce a reduction of 0.081 of a car theft per month. The average value of the stolen cars in this sample is $8,403. Thus, in terms of the reduction in auto-theft exclusively, the protection policy was not cost-effective.

20

Policy Discussion

•

Of course, the deployment of visible police protection could also deter other types of crime, and benefit citizens in terms of increased feelings of security. Importantly, incapacitating criminals is likely to substantially benefit society, whereas the estimates of this paper capture only deterrence effects.

•

On the other hand, part of the car theft reduced in the protected blocks could just be displaced to other areas (or to other criminal activities).

21

Conclusions •

• •

The estimates of this paper suggest that there is a large, negative, but very local effect of observable police presence on car theft. The magnitude of the same-block effect is important in economic terms (75%). This effect is due to deterrence (not incapacitation). The paper finds no effect of observable police presence on car theft in the immediate surrounding area (one, two, or more blocks away). However, these results are pessimistic on the social returns of this police technology: the deployment of uniformed police officers in fixed locations

22

Example

The Central Role of Noise in Evaluating Interventions that use Test Scores to Rank Schools Chay et al. (2005) AER 23

Hyphotetical Program Assignment and Effects on Test Scores

24

P-900 Effects on 1988-1990 Math Gain Scores Panel A: Mathematics P-900

(1)

(2)

2.28*** (0.40)

-0.02 (0.47)

Score relative to cutoff

1988-1990 Gain score (3) (4) -0.11 (0.46)

-0.16 (0.51)

(5) 0.25 (0.53)

-0.16*** (0.02)

σ2λ

142.32*** (18.36)

SES index, 1990

0.15*** (0.01)

Cubic in 1988 score

N

N

N

Y

Y

Region dummies

N

N

N

N

Y

0.013

0.041

0.046

0.041

0.130

2,644

2,644

2,644

Adjusted R2

Sample size 2,644 2,644 Huber-White standard errors are in parentheses. *** significant at 1% ** significant at 5% * significant at 10%

25

P-900 Effects on 1988-1992 Math Gain Scores Panel A: Mathematics P-900

1988-1992 Gain score (8)

(6)

(7)

3.74*** (0.44)

1.61*** (0.50)

Score relative to cutoff

1.48*** (0.48)

(9)

(10)

1.79*** (0.56)

2.09*** (0.60)

-0.15*** (0.02)

σ 2λ

141.65*** (34.01)

SES index, 1990

0.18*** (0.01)

Change in SES, 1990-1992

0.07*** (0.01)

Cubic in 1988 score

N

N

Y

Y

Y

Region dummies

N

N

N

N

Y

0.031

0.053

0.060

0.053

0.140

2,591

2,591

2,591

Adjusted R 2

Sample size 2,591 2,591 Huber-White standard errors are in parentheses. *** significant at 1% ** significant at 5% * significant at 10%

26

P-900 Effects on 1988-1990 Language Gain Scores Panel B: Language P-900

(1)

(2)

4.25*** (0.39)

0.25 (0.44)

Score relative to cutoff

1988-1990 Gain score (3) (4) 0.18 (0.41)

-0.02 (0.48)

(5) 0.54 (0.49)

-0.28*** (0.02)

σ2λ

68.79*** (5.55)

SES index, 1990

0.13*** (0.01)

Cubic in 1988 score

N

N

N

Y

Y

Region dummies

N

N

N

N

Y

0.050

0.147

0.151

0.155

0.230

2,644

2,644

2,644

Adjusted R2

Sample size 2,644 2,644 Huber-White standard errors are in parentheses. *** significant at 1% ** significant at 5% * significant at 10%

27

P-900 Effects on 1988-1992 Language Gain Scores Panel B: Language P-900

1988-1992 Gain score (8)

(6)

(7)

5.94*** (0.39)

2.24*** (0.44)

Score relative to cutoff

2.09*** (0.43)

(9)

(10)

1.67*** (0.48)

2.10*** (0.52)

-0.26*** (0.02)

σ 2λ

62.32*** (11.21)

SES index, 1990

0.16*** (0.01)

Change in SES, 1990-1992

0.07*** (0.01)

Cubic in 1988 score

N

N

Y

Y

Y

Region dummies

N

N

N

N

Y

0.089

0.163

0.175

0.173

0.250

2,591

2,591

2,591

Adjusted R2

Sample size 2,591 2,591 Huber-White standard errors are in parentheses. *** significant at 1% ** significant at 5% * significant at 10%

28

P-900 Effects on 1988-1992 Gain Scores, within Narrow Bands of the Selection Threshold + 5 Points Panel A: Mathematics P-900

+ 3 Points

+ 2 Points

(1)

(2)

(3)

(4)

(5)

(6)

1.50** (0.60)

1.82*** (0.66)

1.79*** (0.73)

2.00*** (0.77)

2.37*** (0.84)

2.39*** (0.85)

SES index, 1990

0.14*** (0.02)

0.13*** (0.03)

0.12*** (0.03)

Change in SES, 1990-1992

0.08*** (0.02)

0.09*** (0.02)

0.06*** (0.02)

Cubic in 1988 score R2

N

Y

N

Y

N

Y

0.007

0.067

0.011

0.074

0.021

0.080

553

363

363

Sample size 883 883 553 Huber-White standard errors are in parentheses. *** significant at 1% ** significant at 5% * significant at 10%

29

P-900 Effects on 1988-1992 Gain Scores, within Narrow Bands of the Selection Threshold + 5 Points Panel B: Language P-900

+ 3 Points

+ 2 Points

(1)

(2)

(3)

(4)

(5)

(6)

2.78*** (0.54)

2.23*** (0.57)

2.10*** (0.69)

1.96*** (0.70)

2.62*** (0.80)

2.48*** (0.75)

SES index, 1990

0.13*** (0.02)

0.12*** (0.03)

0.12*** (0.03)

Change in SES, 1990-1992

0.07*** (0.02)

0.09*** (0.02)

0.06*** (0.02)

Cubic in 1988 score R2

N

Y

N

Y

N

Y

0.030

0.111

0.017

0.101

0.029

0.111

553

363

363

Sample size 883 883 553 Huber-White standard errors are in parentheses. *** significant at 1% ** significant at 5% * significant at 10%

30