Data Analysis and Statistical Methods Statistics 651 http://www.stat.tamu.edu/~suhasini/teaching.html
Lecture 26 (MWF) Testing equality of means of multiple populations - the ANOVA
Suhasini Subba Rao
Lecture 26 (MWF) Analysis of variance - ANOVA
• Formally we want to test H0 : µ1 = µ2 = µ3. against HA : The means are not all the same. How can we do the test? • One method would be to go through every combination and test individually H0 : µ1 = µ2 against HA : µ1 6= µ2 H0 : µ1 = µ3 against HA : µ1 6= µ3 H0 : µ2 = µ3 against HA : µ2 6= µ3. • There are problems associated with doing multiple tests, one of the main is the false discover or false positives.
Lecture 26 (MWF) Analysis of variance - ANOVA
Motivations for the ANOVA • We defined the F-distribution, this is mainly used in the ANOVA.
Suppose we have three samples from three populations:
• Motivating the ANOVA
– 13 heights of 10 year old children from Country 1. – 15 heights of 10 year old children from Country 2. – 18 heights of 10 year old children from Country 3. • We think (to make everything easy for now), that all the observations are normal and we want to test whether they have they same mean. • Let µ1= mean height of 10 year olds in Country 1. µ2= mean height of 10 year olds in Country 2 and µ3= mean height of 10 year olds in Country 3. 1
Lecture 26 (MWF) Analysis of variance - ANOVA
Example 1: The speed of light data • In 1879, A. A. Michelson wanted to measure the speed of light. He conducted 5 trials and for each trial made 20 measurements. Eg. he has 5 samples, where each sample is of size 20. Each sample comes from a different population.
Trial 1 909 11009
Trial 2 856 3741
Trial 3 845 6257
Trial 4 820.5 3605
Trial 5 831.5 2939
• Here are the summary statistics (to make life easy we have removed 299,000 from each observation):
sample mean sample variance
• It is now known that the speed of light (minus 299,000km/s) is 734.5km/s. 3
• In the case of testing equality of means we often use what is known as ANOVA, whereas above we had to so a multiple test this is just one test. 2
Lecture 26 (MWF) Analysis of variance - ANOVA
2
3
4
5
Before we do tests using this data let us look at a simple example. 5
A boxplot for each trial. What do you think? Let µ1, µ2, µ3, µ4, µ5 be the population means of Trial 1,2,3,4,5 respectively. We want to test the hypothesis H0 : µ1 = µ2 = µ3 = µ4 = µ5.
1
Boxplots of the speed of light data
Lecture 26 (MWF) Analysis of variance - ANOVA
• If we do a hypothesis test for each trial and test whether the population mean for each trial is 734.5km/s, we would reject the null. For practice you can try doing the test, and see what you get. • Which indicates there were some problems in his experiment (quite a lot of measurement error). • Another interesting question is whether all trials have the same mean or whether some are different. There is a belief that he changed some of the equipment for each of the trials. By changing the equipment he has produced a new population. • Translating this into statistical language we can ask data all have the same population mean. • First we need to make some plots. 4
1000 900 800 700
Lecture 26 (MWF) Analysis of variance - ANOVA
Example 2: Does hormone treatment has an influence on the weight of 5 week old calves • Upload the calf weight data into JMP. At birth each half was randomly assigned to one of 4 different treatments - the treatment they were given in in the TRT column. There are approximately 10 calves in each treatment group. • The question of interest is whether the treatment has an influence on their weight at 5 weeks. Looking at the summary statistics, we see that the sample means for each groups are different. • We know that the sample means are highly likely to be different because of the different samples lead to different sample means.
Lecture 26 (MWF) Analysis of variance - ANOVA
difference in the population mean weights would suggest that different treatments have an influence on the weight at 5 weeks.
7
• What we want to know is there an actual difference in mean weights over and above the variation due to sample (random variation). A real 6
Lecture 26 (MWF) Analysis of variance - ANOVA
Combined Sample
4.98 0.26
s2
s1
=
=
=
2
s3 =
2
2
2
sA =
• Do you think there is a difference between the three population means?
1 2 2 2 2 ((4.1 − 3.2) + (3.3 − 3.2) + (2.6 − 3.2) + (2.8 − 3.2) ) = 0.45 3 1 2 2 2 ((5.1 − 5.13) + (5 − 5.13) + (5.3 − 5.13) )) = 0.023 2 1 2 2 2 2 ((6.6 − 6.65) + (6.2 − 6.65) + (7.3 − 6.65) + (6.5 − 6.65) ) = 0.22 4 1 2 2 ((4.1 − 3.2) + . . . + (6.5 − 6.65) ) 8 1 2 2 2 (3 × s1 + 2 × s2 + 3 × s3) = 0.26 8
• Notice the in the table we have the means for each group and the last average is overall average. The variances are calculated using:
• Import this data into JMP, look at the plot for each sample.
Lecture 26 (MWF) Analysis of variance - ANOVA
The principles of ANOVA: Dummy example
Sample 2 5.1 5.0 5.3 5.13 0.023
Sample 3 6.6 6.2 7.3 6.5 6.65 0.22
• We do the following example by hand, just so that you have some idea of what is happening. However, in practice an ANOVA should be done on a computer, there is no benefit in learning the mechanical calculations.
Sample 1 4.1 3.3 2.6 2.8 3.2 0.45
• Consider the following dummy example:
average sample variance
9
4
5
6
Graphical representation of data
3
.
7
8
2
Lecture 26 (MWF) Analysis of variance - ANOVA
1
Group mean Total mean
11
Lecture 26 (MWF) Analysis of variance - ANOVA
• We want to test H0 : µ1 = µ2 = µ3 (the population means are the same) against HA at least one of the means are different.
10
2
Lecture 26 (MWF) Analysis of variance - ANOVA
The idea of ANOVA
3
4
.
5 7
Sample 1 3.2 0.45 -1.78 4 × 1.782
Sample 3 6.65 0.22 1.67 4 × 1.672
Combined 4.98
Lecture 26 (MWF) Analysis of variance - ANOVA
Sample 2 5.13 0.023 0.015 3 × 0.0152
The methods of ANOVA mean within variances mean - combined mean ni×(mean - combined mean)2
• We evaluate Sum of Squares Between the samples (SSB): SSB = 4 × 1.782 + 3 × 0.0152 + 4 × 1.672 = 23.8 So SSB/(3 − 1) = 11.9 can almost (BUT IT IS NOT) be considered as the sample variance of the sample means 3.2, 5.13, 6.65. 13
6
• The idea of ANOVA is to look at the variation under the global sample (average calculated using all of the data) and compare it to the variation between the sample means.
1 Within variance Between group variance
12
Lecture 26 (MWF) Analysis of variance - ANOVA
– The within group variation by an average of the above brackets.
Lecture 26 (MWF) Analysis of variance - ANOVA
• Of course we need a test statistic to actually test this.
• We evaluate the Sum of Squares within the samples (SSW): SSW = (4 − 1) × 0.45 + (3 − 1) × 0.023 + (4 − 1) × 0.22 = 2.05. The F-distribution and ANOVA
• We call the 2 and 8 in
SSB/2 SSW/8 ,
SSB/2 SSW/8
∼ F2,8!
the ‘degrees of freedom’.
• The ratio is 11.9/0.266 ≈ 45. And P (F2,8 ≥ 45) ≈ 0, this is very small, hence there is enough evidence to reject the null!
• Under the null H0 : µ1 = µ2 = µ3, we have
SSB/2 , under the null the ratio should be • We consider the ratio F = SSW/8 close to one (hmm this looks rather like the F-test...).
So SSW/[(4 − 1) + (3 − 1) + (4 − 1)] = 0.266 can basically considered as an extended version of the pooled variance, to more than two samples. • If the means of the populations are the same, then SSB/(3 − 1) and SSW/8 should be close to each other. • If the population means are different then, the between group variation must be much greater than the within group variation. • Look at the picture on two pages before:
15
– The between group variation is represented by the lowest bracket. 14
Lecture 26 (MWF) Analysis of variance - ANOVA
Mean square
45
F
very small!!.
Sig.
17
where µ is the global average of the means that is µ = 13 (µ1 + µ2 + µ3).
¢ 1 ¡ SSB 4 × (µ1 − µ)2 + 3 × (µ2 − µ)2 + 4 × (µ1 − µ)2 , ≈ σ2 + 3−1 3−1
• The SSB/2 is basically an estimate of the variance plus additional terms
SSW ≈ σ2 11 − 3
• The SSW/(11 − 3) is basically an estimate of the variance that is
What exactly the SSW and SSB are estimating
Lecture 26 (MWF) Analysis of variance - ANOVA
df
11.9 0.266
The ANOVA table - for the dummy example
2 8 10
• Suppose the means of the populations in this example are µ1, µ2 and µ3. The underlying assumption for the ANOVA is that the variance of all three populations are the same, let us call this σ 2.
Sum of Squares SSB = 23.8 SSW=2.05 25.85
• When we divide the sum of squares by the degrees of freedom this is known as the mean square.
Between Groups Within Groups Total The total = SSW + SSB.
16
Lecture 26 (MWF) Analysis of variance - ANOVA
• We see if the null is true, that is µ1 = µ2 = µ3, then SSB 3−1 is an estimator SSB of σ 2. Hence SSW 11−3 and 3−1 are both estimators of the variance. • However if the alternative is true, and at least one of the means is 2 different, then on average SSB 3−1 will be larger than σ . So the ratio SSB SSW SSB SSW 3−1 / 11−3 will be much bigger than one. This is why when 3−1 / 11−3 is large we reject the null because it is highly unlikely that the means are the same.
Trial 1 909 11009
Trial 2 856 3741
Lecture 26 (MWF) Analysis of variance - ANOVA
Trial 3 845 6257
Trial 4 820.5 3605
Trial 5 831.5 2939
ANOVA for Example 1 (speed of light example) sample mean sample variance • We need to calculate the SSB and SSW, divide each of these by the number of degrees of freedom and compare them to each other and use the F-distribution to determine whether they are close or not. • We use the sample variances to calculate the SSW. Note that the sample size in each sample is 20. SSW = (20 − 1) × (11009 + 3741 + 6257 + 3605 + 2939) = 27551.
19
• To calculate the SSW we need to calculate the distance from each group 18
Lecture 26 (MWF) Analysis of variance - ANOVA
˘ 2 2 2 2 20 × (909 − 852.4) + (856 − 852.4) + (845 − 852.4) + (820.5 − 852.4) 2¯ = 4725.7
(820.5 − 852.4)
5−1 100 − 5 99
df
4725.7/4 = 1181 27551/95 = 290
Mean square
4.1
F
0.004
Sig.
21
• You see that the ratio 1181/290 = 4.1 is quite far from zero, and the p-value is 0.004. Hence at the 5% level, since 0.004 < 0.05, there is evidence to reject the null. That is based on the data, at least one of means for each trial 1,2,3,4,5 is different from the rest.
Lecture 26 (MWF) Analysis of variance - ANOVA
sample mean to the total sample mean. The total sample mean is
=
¯ = (909 + 856 + 845 + 820.5 + 831.5) = 852.4 X 5
SSB
Sum of Squares SSB = 4725.7 SSW= 27551 32276.7
• We need to the make an ANOVA table:
Between Groups Within Groups Total 20
Lecture 26 (MWF) Analysis of variance - ANOVA
23
For example, Xi,j the height of the jth randomly selected child in country i.
X2,1 is the first draw from population two.
X1,2 is the second draw from population one.
X1,1 is the first draw from population one.
• We have k populations with means µ1, . . . , µk . Xi,j denotes an observations from the ith population. In other words Xi,j is the jth draw from the ith population:
ANOVA: The Formal definition that you may have seen in a textbook
Lecture 26 (MWF) Analysis of variance - ANOVA
Doing ANOVA in JMP • It is important to be able to do an ANOVA in JMP - see my JMP help sheet to know of the mechanics of this. • Re-do the above speed of light example in JMP. • Once we have identified that there is a difference, it is of interest to see whether this difference lies, which population (or populations) have a different mean. • Now do the calf weight example in JMP.
22
Lecture 26 (MWF) Analysis of variance - ANOVA
We assume that Xi,j satisfies the following model:
εi,j
= µi + (Xi,j − µi) | {z } = 0 and µ is the common mean.
= µi + εi,j = µ + αi + εi,j k αk
P
Xi,j
where We are testing H0 : µ1 = µ2 = . . . = µk against the alternative HA : the means are not all the same, where sample j is of size nj . Using the notation in the above model this is the same as H0 : α1 = α2 = . . . = αk against alternative HA : at least one of αj s are not zero. • From each population i we have a sample X1,i, . . . , Xni,i, each sample is of size ni.
Sample 1 X1,1 X1,2 .. X1,n1 ¯2 X
Sample 2 X2,1 X2,2 .. X2,n2 ...
... ... ... ... ... ¯ k· X
Sample k Xk,1 Xk,2 .. Xk,nk ¯ X
Global
Lecture 26 (MWF) Analysis of variance - ANOVA
¯1 X
• The data looks like
mean Pni 1 ¯ • The group average X i = ni j=1 Xi,j and global average taken over all ¯ = 1 Pk Pni Xi,j . observations is X i=1 j=1 N
Mean square
F
Sig.
25
• Let N = n1 + . . . + nk . • We use as the test statistic
df
SSB/(k − 1) SSW/(N − k)
∼ Fk−1,N −k .
k−1 N −k N −1
SSB/k−1 SSW/(N −k)
SSB/(k − 1) . SSW/(n − k) • Under the null
Sum of Squares SSB SSW TSS
p-value.
24
(ni − 1)si2, Between Groups Within Groups Total
SSB/(k−1) SSW/(N −k)
Lecture 26 (MWF) Analysis of variance - ANOVA
Calculating the SSW and SSB
¯ i − X) ¯ 2. ni(X
k X i=1
¯ i·)2 which is the sample variance of −X
27
Lecture 26 (MWF) Analysis of variance - ANOVA
• The total sum of all observations is N = n1 + . . . + nk .
i=1
k X
• We calculate the sum of squares between samples SSB =
ni k X X
Pni j=1 (Xi,j
¯ i )2 = (Xi,j − X
• We calculate the sum of squares within sample SSW =
1 ni −1
i=1 j=1
where si2 = sample i. 26
