Analysis of Variance
Why ANOVA? Example: Heart performance scores for 3 groups of subjects, 1=Non-smoker, 2=Moderate smoker, 3=Heavy smoker
Analysis of Variance (ANOVA) Comparing More Than 2 Means
Average
1
2
3
5.90
5.51
5.01
5.92
5.50
5.00
5.89
5.50
4.99
5.91
5.49
4.98
5.88
5.50
5.02
5.90
5.50
5.00
1
2
One-Way ANOVA F-Test When comparing three independent random samples with two-sample t-test, three t-tests would be needed.
One-Way ANOVA F-Test
(Group 1, Group 2), (Group 2, Group 3), (Group 1, Group 3)
Each test suffers a Type I Error rate at α level. If three tests were used simultaneously, the Type I Error will be inflated. One approach to test the significant difference between several means using one single test without inflating Type I Error is Analysis of Variance. 4
3
One-Way ANOVA F-Test Completely Randomized Design
1. Tests the Equality of 2 or More (k) Population Means (μ1=μ2= …=μk ) 2. Variables
One Categorical Independent Variable
Homogeneous Subjects or Experimental Units Are Assigned Randomly to Treatments
2 or More (k) Treatment Levels or Groups
One Quantitative Dependent Variable
3. Used to Analyze Completely Randomized Experimental Designs 5
6
1
Analysis of Variance
One-Way ANOVA F-Test Assumptions
One-Way ANOVA F-Test Hypotheses H0: μ1 = μ2 = μ3 = ... = μk
1. Randomness & Independence of Errors
Independent Random Samples are Drawn
2. Normality
Populations are Normally Distributed
HΑ: Not All μj’s Are Equal
x
μ1 = μ 2 = μ 3 f(x)
At Least one pair of Pop. Means is Different Has Treatment Effect Not imply μ1 ≠ μ2 ≠ ... ≠ μk
3. Homogeneity of Variance (σ1=σ2= …=σk )
f(x)
All Population Means are Equal No Treatment Effect
Populations have Equal Variances
x
μ1 = μ 2 μ3
7
8
Why is it called Analysis of Variances? Example: HP for three groups of subjects
Case I
CASE II
1
2
3
2
3
5.90
5.51
5.01
5.90
6.31
4.52
5.92
5.50
5.00
4.42
3.54
6.93
5.89
5.50
4.99
7.51
4.73
4.48
5.91
5.49
4.98
7.89
7.20
5.55
8
5.8 7
5.6
CASE2
1
Case II
6.0
CASE1
CASE I
5.4
6
5 5.2
4 5.0
4.8
5.88
5.50
5.02
3.78
5.72
3.52
0.0
3 1.0
2.0
3.0
4.0
0.0
1.0
2.0
GROUPID
Average
5.90
5.50
5.00
5.90
5.50
3.0
5.00 9
Case 2 4.0
6.0
4.0
GROUPID
10
Why Variances?
Group 1 Group 2 Group 3
8.0 Pop 1 Pop 2 Pop 3
High BETWEEN variability
Case 1 Pop 4
5.0
5.5
Pop 5
s B2
→ σ B2
Pop 6
High WITHIN variability
6.0
sW2 → σW2 11
12
2
Analysis of Variance
One-Way ANOVA Basic Idea
sB2 sW2
Heart Performance Example Step: 1
1. Compares 2 Types of Variation to Test Equality of Means Based on Ratio of Variances
Hypothesis: H0 : μ1 = μ2 = μ3
(All means are equal.)
2. If Treatment Variation Is Significantly Greater Than Random Variation then Means Are Not Likely Equal
HA : μi ≠ μj for at least one pair of (i, j), i, j = 1, 2, 3. (Not all means are equal.)
13
n = 15 n1 = n2 = n3 = 5
Notations xij xi
: : si : x : n :
Estimation of Variances Between Group Variance
the j-th element from the i-th group the i-th group mean the i-th group standard deviation the overall sample mean the total sample size (n1 + n2 + … + nk)
sB2 = =
2
= 15
6.0
2
(5 − 1).00025 + (5 − 1).00005 + (5 − 1).00025 = .000183 15 − 3
16
One-Way ANOVA F-Test Critical Value
Test Statistic
If there no significant difference between means, then
~ F (df1, df2) distribution (under H0 )
Reject H0
s 2 F = B2 sW
sB2 is Mean Square Between (MSB ) sW2 is Mean Square for Error (MSW )
Do Not Reject H0
would be a small value.
Degrees of Freedom
5.5
2
2 W
One-Way ANOVA F-Test Test Statistic
df1 = k – 1 df2 = n – k
6.0
2
5(5.9 − 5.47) + 5(5.5 − 5.47) + 5(5 − 5.47) = 1.46 3 −1 5.0
x35
5.5
2
( n − 1) s1 + ( n2 − 1) s2 + ... + ( nk − 1) sk s = 1 n−k
x11 = 5.90, s11 = .0158 Group 2: 5.51 5.50 5.50 5.49 5.50 x22 = 5.50, s 22 = .0071 Group 3: 5.01 5.00 4.99 4.98 5.02 x33 = 5.00, s33 = .0158 Group 1: 5.90 5.92 5.89 5.91 5.88
sB2 sW2
5.0
n1 ( x1 − x ) + n2 ( x2 − x ) + ... + nk ( xk − x ) k −21 2 2 2
Within Group Variance
x11 x12 x13
F=
14
F
0
(Numerator Degrees of Freedom) (Denominator Degrees of Freedom)
α
Fα (k – 1, n –k)
k = # Populations, Groups, or Levels n = Total Sample Size
Only reject H0 if having large F ! 17
18
3
Analysis of Variance
Birth Weight Example
Decision Rule & Conclusion
Step: 2
Test Statistic: F =
Step: 3
2 B 2 W
s 1.46 = = 7978.1 s 0.000183
Decision Rule:
Example: The birth weight of an infant has been hypothesized to be associated with the smoking status of the mother during the first trimester of pregnancy. The mothers are divided into four groups according to smoking habit, and the sample of birth weights in pounds within each group is given as follow:
Critical Value Approach: Reject null hypothesis, if F > F.05 , 2, 12 = 3.89 . (Table 5, page A-12.) p-value Approach: Reject null hypothesis if p -value is less than .05.
Step: 4
Conclusion: The test statistic F = 7978.1 > 3.89, and since F.001, 2, 12 = 12.97 < 7938.8, so p -value < 0.05, null hypothesis is rejected. There is statistically significant difference between group means. 19
Birth Weight Example
20
Example using sample variances
(Observational Study)
Group 1: (Mother is a nonsmoker) 7.5 6.9 7.4 9.2 8.3 7.6 Group 2: (Mother is an ex-smoker but not during the pregnancy) 5.8 7.1 8.2 7.1 7.8 Group 3: (Mother is a current smoker and smoke less than 1 pack per day) 5.9 6.2 5.8 4.7 7.2 6.2 Group 4: (Mother is a current smoker and smoke more than 1 pack per day) 6.8 5.7 4.9 6.2 5.8 5.4 6.2
Hypothesis: H0 : μ1 = μ2 = μ3 = μ4 Ha : μi ≠ μj for at least one pair of (i, j), i, j = 1, 2, 3, 4. Test Statistic:
F=
s B2 sW2
~ F-distribution (df1=3, df2=20)
Over all mean, x = 6.7222 x 2 = 7.2
x 3 = 6.0
x 4 = 5.8571
s1= .8134
x 1 = 7.8167
s2 = .9138
s3 = .8075
s4 = .6161
n1 = 6
n2 = 5
n3 = 6
n4 = 7
21
Decision Rule & Conclusion
Mean Squares for F Test
Test Statistic: F =
Between groups variability:
s B2 =
6( x1 − x ) 2 + 5( x 2 − x ) 2 + 6( x3 − x ) 2 + 7( x 4 − x ) 2 4 −1
(6 − 1)s1 + (5 − 1)s 2 + (6 − 1)s3 + (7 − 1)s 4 2
2
24 − 4 where n = 6 + 5 + 6 + 7 = 24 ,
2
2 W
s B2 5.537 5.508 = = 9.088 sW2 0.609 .634
Decision Rule: (Table 5, page A-13.) Critical Value Approach: Reject null hypothesis, if F > F .05 , 3, 20 = 3.10 . p-value Approach: Reject null hypothesis, if p -value is less than .05.
= 5.537
Within groups variability:
sW2 =
22
Conclusion: The test statistic F = 9.088 > F.05, 3, 20 = 3.10, and p -value is less than .001 since F.001, 3, 20 = 8.10 < 9.088, so p-value < 0.05, the null hypothesis is rejected. There is significant difference between group means.
2
= .609,
s is a good estimate of σ 2. 23
24
4
Analysis of Variance
Error Bar Chart
SPSS Output
SPSS Error Bar Chart 9
p-value
ANOVA
95% CI Birth Weight
8
BWEIGHT Sum of Squares
df
Mean Square
Between Groups
16.611
3
5.537
Within Groups
12.185
20
.609
Total
28.796
23
F
Sig.
9.088
.001
7
6
5
sw2
sB2
4 N=
6
5
6
7
Non smoker
Ex-smoker
Smoker 1
Smoker Categories
25
26
Error Bar Chart 8.00
What’s in the ANOVA Table?
] ]
]
weight
6.00
]
4.00
2.00
1
2
Nonsmoker
Ex-smoker
3
4
Smoke < 1
Smoke > 1
status
28
27
One-Way ANOVA Partitions Total Variation
Total Variation TSS = ( x11 − x )2 + ( x12 − x )2 + K + (xkn − x )2 2
k
What’s in the ANOVA Table? Sum of Squares Between Sum of Squares Treatment
2
kk
= ∑∑ ( xij − x ) 2
Total variation
Variation due to treatment
2
ni
i =1 j =1
TSS = (7.5 – 6.7222)2 + … + (6.2 – 6.7222)2 = 28.796
Variation due to random sampling
ANOVA
Sum of Squares Within Sum of Squares Error
BWEIGHT Sum of Squares
29
df
Mean Square
Between Groups
16.611
3
5.537
Within Groups
12.185
20
.609
Total
28.796
23
F 9.088
Sig. .001
30
5
Analysis of Variance
Treatment Variation
Random (Error) Variation
SSB = n11( x11 − x )2 + n22( x22 − x ) 2 + K+ nkk ( xkk − x ) 2 2
=
2
k
∑ n (x i
2
k
i =1
SSB = 6(7.8167 – = 16.611
2
ni
i =1 j =1
6.7222)2
)
22
k
= ∑∑ ( xij − xi ) 2 = ∑ (ni − 1) si2
− x )2
i
(
SSW = ( x11 − x11) 2 + ( x21 − x22 ) 2 + K + xknknii − xkk 11 21
2
+ 5(7.2 –
6.7222)2
i =1
SSW = (7.5 – 7.8167)2 + … + (6.2 – 5.8571)2 = 12.185
+…
ANOVA
ANOVA
BWEIGHT
BWEIGHT Sum of Squares
df
Mean Square
Between Groups
16.611
3
5.537
Within Groups
12.185
20
.609
Total
28.796
23
F 9.088
Sig.
Sum of Squares
.001
31
One-Way ANOVA Summary Table Source of Sum of Variation Squares Treatment
(Between samples)
Error (Within samples)
Total
SSB
(16.611)
SSW
(12.185)
TSS
(28.796)
Degrees of Freedom k-1 (3)
n-k (20)
Mean Square
16.611
3
5.537
Within Groups
12.185
20
.609
Total
28.796
23
F
Sig.
9.088
.001
32
SPSS Output
Mean F Square (Variance) MSB MSB (5.537) MSW MSW
df
Between Groups
p-value
ANOVA BWEIGHT Sum of Squares
(9.088)
(.609)
df
Mean Square
Between Groups
16.611
3
5.537
Within Groups
12.185
20
.609
Total
28.796
23
F 9.088
Sig. .001
n-1 (23)
* MSB=SSB/(k - 1); MSW=SSW/(n - k); TSS=SSB+SSW
33
What if the assumptions are not satisfied?
34
Multiple Comparisons What should we do if we found significant difference between population means?
Try a nonparametric method: Kruskal-Wallis Test
35
36
6
Analysis of Variance
Tukey’s B
Post Hoc Analysis SPSS output tables from Bonferroni and Tukey’s-b options. Multiple Comparisons
WEIGHT
Dependent Variable: BWEIGHT Bonferroni
Std. Error
(I) SMOKEST
(J) SMOKEST
Lower Bound
Upper Bound
Non smoker
Ex-smoker
.6167
.4727
1.000
-.7668
2.0002
Smoker 1
1.9595*
.4343
.001
.6884
3.2307
Non smoker
-.6167
.4727
1.000
-2.0002
.7668
Smoker 1
1.3429*
.4570
.049
5.020E-03
2.6807
Non smoker
-1.8167*
.4507
.004
-3.1358
Ex-smoker
-1.2000
.4727
.117
-2.5835
.1835
Smoker >1
.1429
.4343
1.000
-1.1283
1.4140
Non smoker
-1.9595*
.4343
.001
-3.2307
-.6884
Ex-smoker
-1.3429*
.4570
.049
-2.6807
-5.0200E-03
Smoker 1 pack Smoke < 1 pack Ex-smoker Nonsmoker
95% Confidence Interval
Mean Difference (I-J)
Sig.
7 6 5 6
Subset for alpha = .05 1 2 5.8571 6.0000 7.2000 7.8167
Means for groups in homogeneous subsets are displayed. a. Uses Harmonic Mean Sample Size = 5.915. b. The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.
-.4975
*. The mean difference is significant at the .05 level.
N
37
Error in Multiple Comparison Procedures
38
Bonferroni Adjustment Method In Multiple Comparisons using Confidence Interval Estimate for difference of two means with sw2 as the pooled estimate of common variance for multiple comparisons and with Bonferroni correction, 1−α∗ is the corrected confidence level, and α∗ is the corrected level of significance when making c comparisons.
Individual error rate: The probability that a comparison of means will be falsely declared significant in an experiment. (αΙ)
Family-wise error rate: The probability that at least a pair of means will be falsely declared significant in an experiment that makes m comparisons. (1–(1–αΙ)m)
Since (1 – (1−α∗)c) ≤ cα∗ = α
⇒
α* =
α c
For having 3 pairs, as the example, the probability of falsely reject at least one pair of mean is about 0.05, if α∗=.05/3 ≈ 0.017, and so (1 – (1−.017)3) ≈ 0.05. 39
Confidence Interval
Bonferroni Adjustment Method ⎛k⎞
Example: (from the previous problem about smoking mothers) For comparing “Smoke < 1” & “Smoke > 1”
k!
Number of pairs to be compared is c = ⎜⎜ ⎟⎟ = ⎝ 2 ⎠ 2! ( k − 2)! So, α*=
α
Number of pairs to be compared
40
α = 0.05, α* ≈ α/c = 0.05/{4!/(2!x[4-2]!)} = .05/6 = .008, sw2 = .609, degrees of freedom = 24 − 4 = 20 tα*/2, df = t.004,20 ≈ 2.9, [If not adjusted, α = 0.05, t0.025, 20 = 2.086.]
.
In confidence interval estimation, 1−α* would be the corrected confidence level.
⎛1 1⎞ 5.8571 − 6.000 ± 2.9 ⋅ .609⎜ + ⎟ ⇒ (−1.402, 1.116) ⎝6 7⎠ This interval does contain zero. It implies the difference between the means of “Smoke < 1” & “Smoke > 1” groups is insignificant.
⎛1 1⎞ x1 − x2 ± tα * ⋅ sw2 ⋅ ⎜⎜ + ⎟⎟ 2 ⎝ n1 n2 ⎠ 41
42
7
Analysis of Variance Multiple Comparisons
Two-sample t -Test
Dependent Variable: BWEIGHT Bonferroni
Example: (from the previous problem about smoking mothers) For comparing “Smoke < 1” & “Smoke > 1” α = 0.05, α* ≈ α/c = 0.05/{4!/(2!2!)} = .05/6 = .008, Critical value: tα*/2 = t.004 ≈ 2.9, (d.f. = 24−4=20) sw2 = .609
t=
5.8571 − 6.000 = −0.329 .609 .609 + 6 7
Ex-smoker
.6167
.4727
Smoker 1
1.9595*
Non smoker
Non smoker
Smoker 1
Since 0.329 > 1.325, so p-value > 0.1 > .008. It implies the difference between the means of “Smoke < 1” & “Smoke > 1” groups is insignificant.
Std. Error
(J) SMOKEST
Ex-smoker
95% Confidence Interval
Mean Difference (I-J)
(I) SMOKEST
Sig.
Lower Bound
Upper Bound
1.000
-.7668
2.0002
.004
.4975
3.1358
.4343
.001
.6884
3.2307
-.6167
.4727
1.000
-2.0002
.7668
Smoker 1
1.3429*
.4570
.049
5.020E-03
2.6807
Non smoker
-1.8167*
.4507
.004
-3.1358
-.4975
Ex-smoker
-1.2000
.4727
.117
-2.5835
.1835
Smoker >1
.1429
.4343
1.000
-1.1283
1.4140
Non smoker
-1.9595*
.4343
.001
-3.2307
-.6884
Ex-smoker
-1.3429*
.4570
.049
-2.6807
-5.0200E-03
Smoker
