Chapter 19 - Sample Surveys PART VI : SAMPLING Dr. Joseph Brennan Math 148, BU

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

1 / 20

Parameters & Statistics

Parameter: A numerical fact about a population. The average age of all eligible voters. The percentage of eligible voters registered to vote. The percentage of adults retired/employed/students. As many populations are too large to feasibly take a consensus, sampling is used to estimate a parameter.

Statistic: A numerical fact about a sample. A statistic is used to estimate a parameter. An investigator knows a statistic and wants to know a parameter.

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

2 / 20

Bias

Recall that bias error is not random but affects the data in a single direction. When a selection procedure is biased, taking a larger sample does not lessen the bias. One is just repeating the same basic mistake on a larger scale.

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

3 / 20

Example: Roosevelt and Landon

In 1936, Franklin Delano Roosevelt and Alfred Landon were in competition for the American presidency. The Literary Digest magazine predicted Roosevelt would poll at 43% on election day. The Literary Digest had accurately predicted the winner of every election since 1916. The 1936 prediction was based upon a sample of 2.4 million voters; an obscenely large opinion poll before the advent of the internet.

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

4 / 20

Example: Roosevelt and Landon

Roosevelt won with a majority 62%. The Literary Digest was off by 19 percentage points! The Literary Digest folded after the 1936 election and is best-remembered today for the circumstances surrounding its demise. Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

5 / 20

Example: Roosevelt and Landon

The Literary Digest compiled their 2.4 million responses (todays typical political survey asks between 500 & 1000 respondents) from a total 10 million questionnaires. The response rate was low; 24%. The 10 million names for the questionnaires were pulled from telephone books and car registrations. In 1936 only 1 in 4 American households had telephones and far less owned a car. Remember that 1936 was the height of the Great Depression. The Literary Digest’s error can be partially attributed to sampling bias. During the depression, in an election split ideologically on economic policy, the Literary Digest’s questionnaire was biased against the poor.

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

6 / 20

Bias Sampling Bias: A bias in which a sample is collected in such a way that some members of the intended population are less likely to be included than others. The bias can lead to an over/underrepresentation of the corresponding parameter in the population. Almost every sample in practice is biased because it is practically impossible to ensure a perfectly random sample.

Non-response Bias: A bias that results when respondents differ in meaningful ways from nonrespondents. Nonresponse is often problem with mail surveys, where the response rate can be very low. With a response rate of 25%, the Literary Digest’s error can be partially attributed to nonresponse bias. Respondents and nonrespondents can differ in ways beyond their willingness to answer a questionnaire. Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

7 / 20

Quota Sampling Quota Sampling: A sampling method in which interviewers are assigned a fixed quota of subjects to interview. Example: An interviewer during the 1948 presidential election is sent to St. Louis to interview 13 subjects: 6 from the suburbs and 7 from the city. 7 men and 6 women. Of the 7 men: 3 are to be under 40 and 4 are to be over 40 years old. 1 is to be black and 6 are to be white. Of the 6 white men; concerning their rent: 1 pays more than $44. 3 pay between $18 and $44. 2 pay less than $18. Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

8 / 20

Example: Truman and Dewey In the 1948 presidential election, the three major polls sampled using quota sampling.

The polls gave Dewey a 5 percentage point lead over Truman; people were very confident of Dewey’s eventual victory.

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

9 / 20

Quota Sampling

Quota Sampling is inherently flawed in that unintentional bias is introduced by the interviewers. The point of a quota sample is to find a sample which accurately represents the populations diversity subject to known parameters. The quotas are an indirect way to force a sampling distribution, though this control can be misguided. There are many factors that may influence the results of a sample besides the ones a survey organization controls for.

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

10 / 20

Example: Truman and Dewey In the 1948 election, the Gallup Poll’s quota sampling had a Republican bias.

For a number of confounding reasons, interviewers chose too many Republicans. Within enough demographic groups, the Republicans were marginally easier to interview. Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

11 / 20

Probability Methods

Probability Methods: Sampling techniques which implements an objective chance process to choose subjects from the population, leaving no discretion to the interviewer. It is possible to compute the chance that any particular individual in the population will get into the sample.

Simple Random Sampling: A sampling technique where selection of individuals is equally likely and drawing for the sample is performed without replacement.

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

12 / 20

Probability Methods National elections have a population (eligible voters) of approximately 200 million individuals. To frame an opinion poll where all individuals have an equally likely chance of being polled is not a simple task. Between 1952 and 1984, Gallup used a technique called Multistage Cluster Sampling:

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

13 / 20

Probability Methods

Gallup’s Multistage Cluster Sampling: Divide the country into four geographical regions, each having similar population sizes. In each region, group together all the population centers of similar size and take a random sample. For election purposes, population centers are divided into wards and wards may be further divided into precincts. Among the chosen population centers, take a random sample of wards and precincts. Among these samples, take a random sample of households.

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

14 / 20

Probability Methods

Interviewers sent to individual households are not given discretion as to who they poll. Interviewers may be instructed to: ”Speak with the youngest male eligible voter, of if none are available, speak with the oldest female eligible voter.” This method of questioning is constructed like a quota sample: it is constructed so that the sample distribution and the population distribution are similar for a certain set of parameters. The difference between this probability method and quota sampling is the lack of discretion an interviewer is allowed.

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

15 / 20

Gallup’s Probability Methods Using Probability Methods, Gallup has been able to reduce sample sizes and increase accuracy:

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

16 / 20

Gallup’s Probability Methods

The best probability methods minimize bias by being impartial and objective in choosing samples. Section 6 of this chapter has an in-depth analysis of the Gallup survey from the presidential election of 1984. The goal of the survey is to infer, from the sample, how the nation will vote. However, the population is much more complicated then just eligible voters, and should be amended to eligible voters who intend to vote.

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

17 / 20

Gallup’s Probability Methods Gallup attempts to screen for nonvoters by having them answer questions: 1) How much thought have you given to the coming November elections? 2) Have you ever voted in this precinct or district? 3) Where do people who live in this neighborhood go to vote? 4) In the previous election did things come up which kept you from voting, or did you happen to vote? For whom? Since not voting can have a negative stigma, the above questions can screen for nonvoters without forcing people to admit that they do not attempt to vote. Question 3 is not multiple choice and asks for a specific location. Polling nonvoters will introduce a bias into your sample. The preferences of likely voters differ from preferences of likely nonvoters. Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

18 / 20

Gallup’s Probability Methods Even with personal interviews, many subjects are missed. Since they tend to be different from the subjects available for the interview a non-response bias is created. This bias is adjusted out by giving more weight to subjects who were available but hard to get.

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

19 / 20

Estimating Parameters

When we infer a parameter (population) from a statistic (sample), our estimate has an error described: estimate = parameter + bias + chance error Chance error is sampling error and is natural and measurable for all samples, as they are only part of the whole population. Bias is non-sampling error as this area arises from the way the sample was drawn and is a systematic error.

Dr. Joseph Brennan (Math 148, BU)

Chapter 19 - Sample Surveys

20 / 20