## Using Randomization Methods to Build Conceptual Understanding in Statistical Inference: Day 1

Using Randomization Methods to Build Conceptual Understanding in Statistical Inference: Day 1 Lock, Lock, Lock, Lock, and Lock Minicourse – Joint Math...
Author: Earl Scott
Using Randomization Methods to Build Conceptual Understanding in Statistical Inference: Day 1 Lock, Lock, Lock, Lock, and Lock Minicourse – Joint Mathematics Meetings Boston, MA January 2012 WiFi: marriotconference, password: 1134ams

Introductions: Name Institution

Schedule: Day 1 Wednesday, 1/4, 4:45 – 6:45 pm 1. Introductions and Overview 2. Bootstrap Confidence Intervals • What is a bootstrap distribution? • How do we use bootstrap distributions to build understanding of confidence intervals? • How do we assess student understanding when using this approach? 3. Getting Started on Randomization Tests • What is a randomization distribution? • How do we use randomization distributions to build understanding of p-values? • How do these methods fit with traditional methods? 4. Minute Papers

Schedule: Day 2 Friday, 1/6, 3:30 – 5:30 pm 5. More on Randomization Tests • How do we generate randomization distributions for various statistical tests? • How do we assess student understanding when using this approach? 6. Connecting Intervals and Tests 7. Technology Options • Short software demonstrations (Minitab, Fathom, R, Excel, ...) – pick two! 8. Wrap-up • How has this worked in the classroom? • Participant comments and questions 9. Evaluations

Why use Randomization Methods?

These methods are great for teaching statistics… (the methods tie directly to the key ideas of statistical inference so help build conceptual understanding)

And these methods are becoming increasingly important for doing statistics. (Attend the Gibbs Lecture tonight!)

It is the way of the past… "Actually, the statistician does not carry out this very simple and very tedious process [the randomization test], but his conclusions have no justification beyond the fact that they agree with those which could have been arrived at by this elementary method." -- Sir R. A. Fisher, 1936

… and the way of the future “... the consensus curriculum is still an unwitting prisoner of history. What we teach is largely the technical machinery of numerical approximations based on the normal distribution and its many subsidiary cogs. This machinery was once necessary, because the conceptually simpler alternative based on permutations was computationally beyond our reach. Before computers statisticians had no choice. These days we have no excuse. Randomization-based inference makes a direct connection between data production and the logic of inference that deserves to be at the core of every introductory course.” -- Professor George Cobb, 2007

Question Can you use your clicker? A. Yes B. No C. Not sure D. I don’t have a clicker

Question How frequently do you teach Intro Stat? A. Regularly B. Occasionally C. Rarely/Never

Question How familiar are you with simulation methods such as bootstrap confidence intervals and randomization tests? A. Very B. Somewhat C. A Little / Not at all

Question Have you used randomization methods in any statistics class that you teach? A. Yes, as a significant part of the course B. Yes, as a minor part of the course C. No

Question Have you used randomization methods in Intro Stat? A. Yes, as a significant part of the course B. Yes, as a minor part of the course C. No

Intro Stat – Revise the Topics • • •• • • • •

Descriptive Statistics – one and two samples Normal distributions Bootstrap confidence intervals Data production (samples/experiments) Randomization-based hypothesis tests Sampling distributions (mean/proportion) Normal/sampling distributions Confidence intervals (means/proportions)

• Hypothesis tests (means/proportions) • ANOVA for several means, Inference for regression, Chi-square tests

It’s close to 5 pm; We need a snack!

What proportion of Reese’s Pieces are Orange? Find the proportion that are orange for your box.

Bootstrap Distributions Or: How do we get a sense of a sampling distribution when we only have ONE sample?

Suppose we have a random sample of 6 people:

Original Sample

Create a “sampling distribution” using this as our simulated population

Bootstrap Sample: Sample with replacement from the original sample, using the same sample size.

Original Sample

Bootstrap Sample

Simulated Reese’s Population

Sample from this “population”

Create a bootstrap sample by sampling with replacement from the original sample. Compute the relevant statistic for the bootstrap sample. Do this many times!! Gather the bootstrap statistics all together to form a bootstrap distribution.

Original Sample Sample Statistic

Bootstrap Sample

Bootstrap Statistic

Bootstrap Sample

Bootstrap Statistic

. . . Bootstrap Sample

. . . Bootstrap Statistic

Bootstrap Distribution

We need technology! Introducing

StatKey.

Example: Atlanta Commutes What’s the mean commute time for workers in metropolitan Atlanta? Data: The American Housing Survey (AHS) collected data from Atlanta in 2004.

Sample of n=500 Atlanta Commutes Dot Plot

CommuteAtlanta

20

40

60

80

100

120

140

160

Time

Where might the “true” μ be?

180

StatKey can be found at

www.lock5stat.com

How can we get a confidence interval from a bootstrap distribution?

Method #1: Use the standard deviation of the bootstrap statistics as a “yardstick”

Using the Bootstrap Distribution to Get a Confidence Interval – Version #1 The standard deviation of the bootstrap statistics estimates the standard error of the sample statistic.

Quick interval estimate :

For the mean Atlanta commute time:

Using the Bootstrap Distribution to Get a Confidence Interval – Version #2 95% CI=(27.35,30.96) Chop 2.5% in each tail

Keep 95% in middle

Chop 2.5% in each tail

For a 95% CI, find the 2.5%-tile and 97.5%-tile in the bootstrap distribution

90% CI for Mean Atlanta Commute 90% CI=(27.64,30.65) Chop 5% in each tail

Keep 90% in middle

Chop 5% in each tail

For a 90% CI, find the 5%-tile and 95%-tile in the bootstrap distribution

Bootstrap Confidence Intervals Version 1 (Statistic ± 2 SE): Great preparation for moving to traditional methods Version 2 (Percentiles): Great at building understanding of confidence intervals

Playing with StatKey! See the purple pages in the folder.

We want to collect some data from you. What should we ask you for our one quantitative question and our one categorical question?

What quantitative data should we collect from you? A. What was the class size of the Intro Stat course you taught most recently?

What categorical data should we collect from you? A. Did you fly or drive to these meetings? B. Have you attended any previous JMM meetings?

How do we assess student understanding of these methods (even on in-class exams without computers)? See the green pages in the folder.

Paul the Octopus

Paul the Octopus • Paul the Octopus predicted 8 World Cup games, and predicted them all correctly • Is this evidence that Paul actually has psychic powers? • How unusual would this be if he were just randomly guessing (with a 50% chance of guessing correctly)? • How could we figure this out?

Simulate! • Each coin flip = a guess between two teams • Heads = correct, Tails = incorrect • Flip a coin 8 times and count the number of heads. Remember this number! Did you get all 8 heads? (a) Yes (b) No

Hypotheses Let p denote the proportion of games that Paul guesses correctly (of all games he may have predicted) H0 : p = 1/2 Ha : p > 1/2

Randomization Distribution • A randomization distribution is the

distribution of sample statistics we would observe, just by random chance, if the null hypothesis were true • A randomization distribution is created by simulating many samples, assuming H0 is true, and calculating the sample statistic each time

Randomization Distribution • Let’s create a randomization distribution for Paul the Octopus! • On a piece of paper, set up an axis for a dotplot, going from 0 to 8 • Create a randomization distribution using each other’s simulated statistics • For more simulations, we use StatKey

p-value • The p-value is the probability of getting a statistic as extreme (or more extreme) as that observed, just by random chance, if the null hypothesis is true • This can be calculated directly from the randomization distribution!

StatKey

( 2) 1

8

= 0.0039

Randomization Test • Create a randomization distribution by simulating assuming the null hypothesis is true • The p-value is the proportion of simulated statistics as extreme as the original sample statistic

Coming Attractions - Friday • How do we create randomization distributions for other parameters? • How do we assess student understanding? • Connecting intervals and tests • Technology for using simulation methods • Experiences in the classroom

Using Randomization Methods to Build Conceptual Understanding of Statistical Inference: Day 2 Lock, Lock, Lock, Lock, and Lock Minicourse- Joint Mathematics Meetings Boston, MA January 2012

Cocaine Addiction • In a randomized experiment on treating cocaine addiction, 48 people were randomly assigned to take either Desipramine (a new drug), or Lithium (an existing drug) • The outcome variable is whether or not a patient relapsed • Is Desipramine significantly better than Lithium at treating cocaine addiction?

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

1. Randomly assign units to treatment groups

Desipramine R

R

R

R

Lithium R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

2. Conduct experiment 3. Observe relapse counts in each group R = Relapse N = No Relapse

1. Randomly assign units to treatment groups

Desipramine R

R

R

R

R

R

R

R

R

R

N R

N R

N R

N R

N

N

N

N

N

N

N

N

N

N

10 relapse, 14 no relapse

pˆ new − pˆ old 10 18 = − 24 24 = −.333

Lithium R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

N

N

N

N

N

N

18 relapse, 6 no relapse

Randomization Test • Assume the null hypothesis is true • Simulate new randomizations • For each, calculate the statistic of interest • Find the proportion of these simulated statistics that are as extreme as your observed statistic

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

N

N

R

R

R

R

R

R

N

N

N

N

N

N

R

R

R

R

R

R

N

N

N

N

N

N

N

N

N

N

N

N

10 relapse, 14 no relapse

18 relapse, 6 no relapse

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

N

N

R

R

R

R

R

R

N

N

N

N

N

N

R

R

R

R

R

R

N

N

N

N

N

N

N

N

N

N

N

N

Simulate another randomization Desipramine

Lithium

R

N

R

N

R

R

R

R

R

R

R

N

R

R

R

N

R

N

N

N

R

R

16 relapse, 8 no relapse

pˆ N − pˆ O 16 12 = − 24 24 = 0.167

N

N

N

R

N

R

R

N

N

N

N

R

N

R

R

N

R

N

R

R

R

R

12 relapse, 12 no relapse

Simulate another randomization Desipramine

Lithium

R

R

R

R

R

R

R

N

R

R

N

N

R

R

N

R

N

R

R

N

R

N

R

R

17 relapse, 7 no relapse

pˆ N − pˆ O 17 11 = − 24 24 = 0.250

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

R

N

N

N

N

N

N

11 relapse, 13 no relapse

Simulate! • Combine everyone into one group, and

rerandomize them into the two groups • Compute your difference in proportions • Create the randomization distribution • How extreme is the observed statistic of -0.33? • Use StatKey for more simulations

The observed difference in proportions was -0.33. How unlikely is this if there is no difference in the drugs? To begin to get a sense of this: How many of you had a randomly simulated difference in proportions this extreme? Was your simulated difference in proportions more extreme than the sample statistic (that is, was it less than or equal to

StatKey

Proportion as extreme as observed statistic

observed statistic

The probability of getting results as extreme or more extreme than those observed if the null hypothesis is true, is about .02.

p-value

Cocaine Addiction You want to know what would happen • Why did you re-deal your cards? • by random chance (the random allocation to treatment groups) • Why did you leave the outcomes (relapse or no relapse) unchanged on each card? • if the null hypothesis is true (there is no difference between the drugs)

How can we do a randomization test for a mean?

Example: Mean Body Temperature Is the average body temperature really 98.6oF?

H0:μ=98.6 Ha:μ≠98.6 Data: A random sample of n=50 body temperatures. Dot Plot

BodyTemp50

96

97

98 99 BodyTemp

100

Data from Allen Shoemaker, 1996 JSE data set article

101

Key idea: Generate samples that are (a) consistent with the null hypothesis (b) based on the sample data.

How to simulate samples of body temperatures to be consistent with H0: μ=98.6?

Randomization Samples How to simulate samples of body temperatures to be consistent with H0: μ=98.6?

Let’s try it on StatKey.

How can we do a randomization test for a correlation?

Is the number of penalties given to an NFL team positively correlated with the “malevolence” of the team’s uniforms?

Ex: NFL uniform “malevolence” vs. Penalty yards

r = 0.430 n = 28 Is there evidence that the population correlation is positive?

Key idea: Generate samples that are (a) consistent with the null hypothesis (b) based on the sample data.

H0 : ρ = 0 r = 0.43, n = 28

How can we use the sample data, but ensure that the correlation is zero?

Randomize one of the variables! Let’s look at StatKey.

Playing with StatKey! See the orange pages in the folder.

Choosing a Randomization Method Example: Word recall A=Sleep

14

18

11

13

18

17

21

9

16

17

14

15 mean=15.25

B=Caffeine

12

12

14

13

6

18

14

16

10

7

15

10 mean=12.25

H0: μA=μB vs. Ha: μA≠μB

Reallocate Option 1: Randomly scramble the A and B labels and assign to the 24 word recalls. Resample Option 2: Combine the 24 values, then sample (with replacement) 12 values for Group A and 12 values for Group B.

Question In Intro Stat, how critical is it for the method of randomization to reflect the way data were collected? A. Essential

How do we assess student understanding of these methods (even on in-class exams without computers)? See the blue pages in the folder.

Collecting More Data from You!

Rock-Paper-Scissors (Roshambo)

Play a game! Can we use statistics to help us win?

Rock-Paper-Scissors Which did you throw? A). Rock B). Paper C). Scissors

Rock-Paper-Scissors Are the three options thrown equally often on the first throw? In particular, is the proportion throwing Rock equal to 1/3?

Intro Stat – Revised the Topics • • • • • •

Data production (samples/experiments) Descriptive Statistics – one and two samples Bootstrap confidence intervals Randomization-based hypothesis tests Normal/sampling distributions Confidence intervals (means/proportions)

• Hypothesis tests (means/proportions) • ANOVA for several means, Inference for regression, Chi-square tests

Transitioning to Traditional Inference AFTER students have seen lots of bootstrap distributions and randomization distributions… Students should be able to • Find, interpret, and understand a confidence interval • Find, interpret, and understand a p-value

Bootstrap and Randomization Distributions Dot Plot

Measures from Scrambled Collection 1

Slope :Restaurant tips

-60

-40

Correlation: Malevolent uniforms

Dot Plot

Measures from Scrambled RestaurantTips

-20 0 20 slope (thousandths)

40

60 -0.6

-0.4

-0.2

0.0 r

0.2

0.4

All bell-shaped What do you Mean :Body Temperatures Diff means: Finger taps distributions! notice? Dot Plot

Measures from Sample of BodyTemp50

98.2

98.3

98.4

98.5

98.6 Nullxbar

98.7

98.8

0.5 phat

0.6

98.9

99.0

-4

Proportion : Owners/dogs

0.4

Dot Plot

Measures from Scrambled CaffeineTaps

-3

-2

-1

0 Diff

1

2

3

4

Dot Plot

Measures from Sample of Collection 1

0.3

0.6

Mean : Atlanta commutes

Dot Plot

Measures from Sample of CommuteAtlanta

0.7

0.8

26

27

28

29 xbar

30

31

32

The students are primed and ready to learn about the normal distribution!

Transitioning to Traditional Inference • Introduce the normal distribution (and later t) • Introduce “shortcuts” for estimating SE for proportions, means, differences, slope…

Confidence Intervals

95%

-z*

z*

Hypothesis Tests

95%

Area is p-value

Test statistic

Yes! Students see the general pattern and not just individual formulas!

Connecting CI’s and Tests Dot Plot

Measures from Sample of BodyTemp50

Randomization body temp means when μ=98.6 98.2

98.3

98.4

98.5

98.6 xbar

98.7

98.8

98.9

99.0

Dot Plot

Measures from Sample of BodyTemp50

Bootstrap body temp means from the original sample 97.9

98.0

98.1

98.2

98.3 98.4 bootxbar

98.5

98.6

98.7

What’s the difference?

Fathom Demo: Test & CI

Sample mean is in the “rejection region”

Null mean is outside the confidence interval

Technology Sessions Choose Two! (The folder includes information on using Minitab, R, Excel, Fathom, Matlab, and SAS.)

Student Preferences Which way did you prefer to learn inference (confidence intervals and hypothesis tests)? Bootstrapping and Randomization

39 67%

Formulas and Theoretical Distributions

19 33%

Student Preferences Which way do you prefer to do inference? Bootstrapping and Randomization

42 72%

Formulas and Theoretical Distributions

16 28%

Student Preferences Which way of doing inference gave you a better conceptual understanding of confidence intervals and hypothesis tests? Bootstrapping and Randomization

42 72%

Formulas and Theoretical Distributions

16 27%

Student Preferences LEARN inference AP Stat No AP Stat

Simulation 13 26

DO inference AP Stat No AP Stat

Simulation 18 24