(Streitfield, D., 1988 and Wallis, 1987)

Recalling some of the information from Example 1.1 in Lesson 1, in 1987, Shere Hite published a best-selling book called Women and Love: A Cultural Revolution in Progress. This 7-year research project produced a controversial 922-page publication that summarized the results from a survey that was designed to examine how American women felt about their relationships with men. Hite mailed out 100,000 fifteen-page questionnaires to women who were members of a wide variety of organizations across the U.S. Questionnaires were actually sent to the leader of each organization. The leader was asked to distribute questionnaires to all members. Each questionnaire contained 127 open-ended questions with many parts and follow-ups. Part of Hite's directions read as follows: "Feel free to skip around and answer only those questions you choose." Approximately 4500 questionnaires were returned.

In Lesson 1, we determined that the

• The population was all American women.
• The sample was the 4,500 women who responded.

It is also easy to identify that the sampling unit was an American woman. So, the key question is "What is the sampling frame?" Some might think that the sampling frame was the 100,000 women who received the questionnaires (that's the intended sample). However, this answer is not correct because the sampling frame was the list from which the 100,000 who were sent the survey was obtained. In this instance, the sampling frame included all American women who had some affiliation with an organization because those are the women that had some possibility of being contacted. If the response rate had been 100%, the sample would have been the 100,000 women who responded to the survey.

You should also remember that ideally, the sampling frame should be as close to the entire population as possible. If this is not possible, the sampling frame should appropriately represent the desired population. In this case, the sampling frame of all American women who were "affiliated with some organization" did not appropriately represent the population of all American women. In Lesson 2, we called this problem selection bias.

This example illustrates three key difficulties that can result in bias in sample surveys:

1. Using the wrong sampling frame. As discussed above, bias can result when the sampling frame leaves out major portions of the population. This is called undercoverage which is a type of selection bias.
2. Not reaching the individuals selected. Because the questionnaire was sent to leaders of organizations, there is no guarantee that these questionnaires actually reached the women who were supposed to be in the sample.
3. Getting a low response rate. In Lesson 1, we learned that this survey has a problem with nonresponse bias because of the low response rate. This problem can create bias if the people who respond have different views than those who do not.

The Gallup Global Emotions Report was released in March 2016 and included the results of surveys Gallup conducted in 140 different countries in 2015 to study the emotional well-being of the populations of each country. For example, Gallup’s survey in Paraguay (population size 7 million) included about 1000 interviews and found the adults in that country to be the happiest in the world with 84% of respondents indicating they had laughed or smiled a lot the day before. On the other hand, Gallup’s survey in Syria (population size 23 million) also included about 1000 interviews and found the adults in that country to be the least happy in the world with only about 36% of respondents indicating they had laughed or smiled a lot the day before. The surveys in both of those two countries had a margin of error of about 3%.

The results of the poll in Paraguay suggest that 84% ± 3% of all Paraguay adults smile or laugh a lot on any given day. What is the correct interpretation of this margin of error?

Margin of Error Interpretation

Assuming the poll in Paraguay used an unbiased procedure, the difference between our sample percent and the true population percent will be within 3%, at least 95% of the time. This means that we are almost certain that 84% ± 3% or (81% to 87%) of all Paraguay adults smile or laugh a lot each day. Because the range of possible values from this poll all fall above 72%, we can also say that we are pretty sure that the rate in Paraguay is above the worldwide average of 72%. If any of the range of possible values would have been 72% or less, then we would not have been able to make that kind of statement with as much certainty. The range of values (81% to 87%) is called a 95% confidence interval. Other levels of confidence, besides 95% may be used - but 95% is the most typical. We will go into further detail about confidence intervals in Lesson 9. Importantly the poll in Syria also had a margin of error of about 3% despite that country having a population that is three times larger. However, the interpretation of the margin of error in Syria should also include the reminder that it reflects only the variability due to the randomness in the survey. The margin of error in that survey does not include any information about the likely bias in the Syria poll that resulted from undercoverage due to the fact that security concerns made it impossible for Gallup to have access to a third of the population (see section 2.1).

Laboratory experiments conducted in the 1980s showed that pregnant mice exposed to high does of ultrasound gave birth to lower weight infant mice than unexposed mice (in fact the higher the dose the greater the effect on birthweights). This worried obstetricians who feared that sonograms given to women during pregnancy might cause lower weights in their children. Researchers at Johns Hopkins University Hospital then examined the birthweights of infants of mothers who had sonograms versus those whose mothers had no such exposure. They found that the 1598 infants who had been exposed averaged a couple of ounces lower in weight than the 944 infants whose mothers did not have a sonogram. However, the women who got sonograms were more likely to have had twins in the past and were more likely to be over 40 years old. Having twins or being over 40 are examples of confounding variables in this study since they provide an alternate explanation for the data. You can not tell whether it was the sonogram that caused the lower birth weights or just the confounding medical reasons for getting the sonogram in the first place. Later experimental evidence in humans did not show sonograms to have any effect (see Abramowicz et al, 2008 for a review).

An educator wants to compare the effectiveness of computer software that teaches reading versus a standard curriculum used to teach reading. The educator tests the reading ability of a group of 60 students and then randomly divides them into classes of 30 students each. One class uses the computer regularly while the other class uses a standard curriculum. At the end of the semester, the educator retests the students and compares the mean increase in reading ability for the two groups.

This example is a randomized experiment because the students were randomly assigned to one of two methods to learn reading.  Also in this example:

• the experimental unit is the student
• the explanatory variable (treatment) is the method used to teach reading
• the response variable is the change found in reading ability at the end of the semester for each individual student.

The randomization that is used in this example cancels out other factors (confounding variables) that could also affect a change in reading ability. Specifically, the randomization will cancel out factors that may result from either self-selected or haphazardly-formed groups. With self-selection, students might base their decision on whether or not they like the computer or whether or not their friends will be in the class. This is no longer a problem when the groups are randomly formed.  Consequently, "cause and effect" statements can be used if statistical significance is found and other precautions are used to treat each group the same except for the different treatments assigned.

In statistics, we also say that the two samples in this study are independent. The label of independent samples is used when the results for the one sample have no impact on the results found in the second sample.  In this instance, each student provided a measurement for only one treatment.  The results from students in one group will not impact the results of students in the other group, so the results from the two samples are independent.

A medical researcher conjectures that smoking can result in wrinkled skin around the eyes. The researcher obtained a sample of smokers and a sample of nonsmokers. Each person was classified as either having or not having prominent wrinkles. The study compared the percent of prominent wrinkles for the two groups.

This example cannot be a randomized experiment because it would be both unrealistic and unethical to randomly assign who would be the smoker and who would be the nonsmoker. Also in this example:

• the experimental unit is the person
• the explanatory variable is smoking status
• the response variable is whether or not each person has prominent wrinkles

Because this example is an observational study, it is possible that confounding variables may also be responsible for whether or not a person has prominent wrinkles. Possible confounding variables include (1) how much time the person spends outside, (2) whether or not the person wears sunscreen, and (3) other variables that revolve around health and nutrition (especially those that could be related to smoking status). Because we can't separate the impact that these variables may have on the response variable, "cause and effect" conclusions are never possible. The researcher would be limited to saying either that there is an association between smoking status and wrinkle status or that there is a difference in the two percents when comparing smokers to nonsmokers.

This is also an example where the two samples are independent. The individuals in this study were classified as being either smokers or nonsmokers. The results from the smoking group had no impact on the results from the nonsmoking group.

Is the right hand stronger than the left hand for those who are right-handed? An instrument has been developed to measure the force exerted (in pounds) when squeezed by one hand. The subjects for this study include 10 right-handed people. How can we best answer this question?

What would happen if we tried to implement what was done in Example 2.4? This would mean that we would randomly assign five people to use their right and five people to use their left hand. The results from the two groups would then be compared. Hopefully, you see that even though randomization is being used with this approach, the results may not be the best because it is possible that - just by the luck of the draw with so few people - the one group could be comprised of strong people while the other group could be comprised of weak people. If this happened, one could erroneously conclude that one hand is stronger for reasons other than that there is a difference in the two hands.

A better approach would be to have each person use both hands and then compare the results for the two hands. With this approach, the

• the experimental unit is the person
• the explanatory variable (treatment) is hand being used (right hand or left hand)
• the response variable is the force exerted (in pounds) for each hand.

The design used in the example is called a block design because the results from each person form a block. Specifically, this block design is called a matched pairs (block) design because each person provides two data observations that can be paired together (i.e. left and right hands of the same person form the pairs). Consequently, we can say that we have two dependent samples. Table 2.4 shows how a spreadsheet for the data in the matched pairs design might look.

In Table 2.4, one sees that the results from each person form a block. The reason that this design is used is so that unwanted or extraneous variation can be removed from the data. In order to accomplish this goal, the data analysis is based on the differences rather than on the original data. By using the differences, we are comparing the two data observations each person provides to each other which distinguishes matched pairs from independent samples. Table 2.5 shows some data that could have been collected in this study.

As you examine the results from Table 2.5, you should see that there are innate differences in strength when comparing the people who participated in the study. For example, Person 1 is much stronger than Person 2. However, the variation from person to person is no longer a factor when the differences are used in the data analysis rather than using the original data.

Also, as you examine Table 2.5, you should see why we classify the two measurements for each experimental unit as the dependent. A higher value in one hand is usually followed by a higher value on the other hand. The values are more similar for each pair of measurements for each experimental unit than the values are between experimental units.

Even though the matched paired design is critical in this example, this study would also benefit from randomization. Since each person is doing both things or providing two measurements, the randomization could be used to determine the order in which the treatments are done. Why would this enhance the study? Problems can exist with block designs, including matched pair designs, when what happens with the first measurement "carries over" to the second measurement. This "carryover" effect is a type of confounding that is found with block designs.

For example, "carryover" effect could possibly occur if complicated equipment was used to measure the force exerted by a hand. If everyone used their right hand first, they might not do so well with the right hand because of not understanding the equipment, but do much better with their left hand because they learned how the equipment worked. In statistics, this is called a training effect. The opposite, however, could also take place. Suppose everyone was asked to first exert force with their right hand for 15 minutes and then repeat this task with their left hand. Participants might do okay with their right hand but become either bored or fatigued or sore when asked to repeat with this task with the left hand. So again, what happened with the first measurement would "carryover" and affect the second measurement. One may conclude that one hand is stronger than the other, not because this is really true, but because the "carryover" effect allowed this to happen.

The overall conclusion is that if you randomly assign the order of treatment, some people will use their right hand first and other people will use their left hand first. This randomization should cancel out the possibility of a "carryover" effect. In statistics, we call this a randomized block design, as shown in Table 2.6. Randomizing the order of treatment makes this a randomized experiment.

An owner of a theater wants to determine if the time of the showing affects attendance at a "scary" movie. In order to check this claim, a sample of five nights from all possible nights over the past month was obtained. The attendance (total number of tickets sold) for both the 7:00 PM and the 9:30 PM showings was determined for each of the five nights.

In this example:

• the experimental unit is the night
• the explanatory variable is the showing time
• the response variable is the attendance at each showing.

This example also uses a matched pair (block) design because there are two measurements made on each night. A picture of this matched pair block design is found in Table 2.4.

Again, why is the matched pairs design preferred over two independent samples? In this example, our goal is to determine whether or not the time of showing affects attendance at the "scary" movie. We do not want any extraneous or other unwanted variation to explain the differences in attendance. In this example, the potential unwanted variation would be the variation that would exist from night tonight. Some of the selected nights may fall on a weekend while other nights may fall on a weekday. This factor could affect attendance. However, this will no longer be a problem when both measurements are made on the same night.

This example, however, cannot be a randomized experiment because it would be impossible to randomly assign the time of showing. The 7:00 PM show will always take place before the 9:30 PM showing. Consequently, there is a possibility that what happens at the 7:00 PM showing may "carryover" and affect attendance at 9:30 PM. A possible "carryover" effect could be the fact there is a limited amount of parking near the theatre. If this were true, perhaps those at the 7:00 PM showing take all the available spots. Then people planning to attend that 9:30 PM showing may not attend because of not being able to find a parking spot. However, this problem may not exist if there is sufficient time between the two showings so that those who attended the 7:00 PM showing had time to leave before those who arrived for the 9:30 PM showing. In any event, because this is an observational study, confounding variables are possible. "Cause and effect" conclusions may not be used if statistical significance is found.

In order to study whether the longterm use of cell phones might be associated with a greater risk of brain tumors, researchers in France conducted a case-control study (see Coureau et al, 2013). In their study 253 glioma patients (a type of brain tumor) were compared as to their cell phone use with 892 matched controls of similar age who lived in the same areas (according to electoral rolls). In this way, the researchers hoped that other environmental factors associated with the differing brain tumor rates seen in differing parts of the country would be eliminated as confounders.

The difficulty in performing a case-control study comes with finding a good group of controls (e.g. similar life circumstances, same gender, similar age, similar similar family histories of disease, etc...).

Notice that case-control studies are done retrospectively - they compare patients who have the disease today with those who don't and ask them about past exposures or behaviors. But people's memories might be faulty or affected by the fact that they have the disease. For example in a study of people who have arthritis, patients will often "remember" that their parents also suffered from arthritis indicating a strong genetic component. However when their non-suffering siblings with the same parents are asked the same question, no genetic component is found.

One alternative is to conduct a Prospective study in which people with different exposures or behaviors (the explanatory variables) are followed over time to see how many in each situation get the disease (the response variable).

447,357 non-Hispanic white cancer free members of the AARP who were 50 to 71 years old in 1995-1996. These subjects were followed for about ten years until the end of 2006 and during that decade 2,904 of them developed melanoma skin cancer. All of the subjects filled out a series of questionnaires at the start of the study and explanatory factors identified at that time could be examined to see if they might be associated with the onset of melanoma later in life. In one study based on this very large cohort of AARP members, Loftfield et al, 2015 found a relationship between heavy coffee drinking (4 cups a day or more) and a modest increase in the risk of getting melanoma. Of course, this observed association can not be viewed as causal since other variables associated with melanoma risk (like exposure to sunlight) might also be different between the heavy coffee drinkers and the non coffee drinkers.