2.6 - Types of Research Studies

So far we have discussed two basic types of comparative statistical research studies: Section

  • experiments/randomized experiments
  • observational studies

With an experiment/randomized experiment, the researcher

  • creates differences in the explanatory variable when assigning/randomly assigning treatments
  • allows for possible "cause and effect" conclusions if other precautions are taken with random assignment of the explanatory variable providing much stronger evidence for “cause and effect” conclusions.
  • when the explanatory variable is randomly assigned, the researcher can minimize the effect of "confounding" variables

With an observational study, the researcher

  • observes differences in the explanatory variable in natural settings/groupings (no variable is randomly assigned)
  • strives for association conclusions since "cause and effect" conclusions are not possible
  • must accept that confounding variables are potential problems

Example 2.4: Randomized Experiment (Two Independent Samples) Section

An educator wants to compare the effectiveness of computer software that teaches reading versus a standard curriculum used to teach reading. The educator tests the reading ability of a group of 60 students and then randomly divides them into classes of 30 students each. One class uses the computer regularly while the other class uses a standard curriculum. At the end of the semester, the educator retests the students and compares the mean increase in reading ability for the two groups.

This example is a randomized experiment because the students were randomly assigned to one of two methods to learn reading.  Also in this example:

  • the experimental unit is the student
  • the explanatory variable (treatment) is the method used to teach reading
  • the response variable is the change found in reading ability at the end of the semester for each individual student.

The randomization that is used in this example cancels out other factors (confounding variables) that could also affect a change in reading ability. Specifically, the randomization will cancel out factors that may result from either self-selected or haphazardly-formed groups. With self-selection, students might base their decision on whether or not they like the computer or whether or not their friends will be in the class. This is no longer a problem when the groups are randomly formed.  Consequently, "cause and effect" statements can be used if statistical significance is found and other precautions are used to treat each group the same except for the different treatments assigned.

In statistics, we also say that the two samples in this study are independent. The label of independent samples is used when the results for the one sample have no impact on the results found in the second sample.  In this instance, each student provided a measurement for only one treatment.  The results from students in one group will not impact the results of students in the other group, so the results from the two samples are independent.

Example 2.5: Observational Study (Two Independent Samples) Section

A medical researcher conjectures that smoking can result in wrinkled skin around the eyes. The researcher obtained a sample of smokers and a sample of nonsmokers. Each person was classified as either having or not having prominent wrinkles. The study compared the percent of prominent wrinkles for the two groups.

This example cannot be a randomized experiment because it would be both unrealistic and unethical to randomly assign who would be the smoker and who would be the nonsmoker. Also in this example:

  • the experimental unit is the person
  • the explanatory variable is smoking status
  • the response variable is whether or not each person has prominent wrinkles

Because this example is an observational study, it is possible that confounding variables may also be responsible for whether or not a person has prominent wrinkles. Possible confounding variables include (1) how much time the person spends outside, (2) whether or not the person wears sunscreen, and (3) other variables that revolve around health and nutrition (especially those that could be related to smoking status). Because we can't separate the impact that these variables may have on the response variable, "cause and effect" conclusions are never possible. The researcher would be limited to saying either that there is an association between smoking status and wrinkle status or that there is a difference in the two percents when comparing smokers to nonsmokers.

This is also an example where the two samples are independent. The individuals in this study were classified as being either smokers or nonsmokers. The results from the smoking group had no impact on the results from the nonsmoking group.

Example 2.6: Experiment/Randomized Experiment (Two Dependent Samples or Matched Pairs) Section

Is the right hand stronger than the left hand for those who are right-handed? An instrument has been developed to measure the force exerted (in pounds) when squeezed by one hand. The subjects for this study include 10 right-handed people. How can we best answer this question?

What would happen if we tried to implement what was done in Example 2.4? This would mean that we would randomly assign five people to use their right and five people to use their left hand. The results from the two groups would then be compared. Hopefully, you see that even though randomization is being used with this approach, the results may not be the best because it is possible that - just by the luck of the draw with so few people - the one group could be comprised of strong people while the other group could be comprised of weak people. If this happened, one could erroneously conclude that one hand is stronger for reasons other than that there is a difference in the two hands.

A better approach would be to have each person use both hands and then compare the results for the two hands. With this approach, the

  • the experimental unit is the person
  • the explanatory variable (treatment) is hand being used (right hand or left hand)
  • the response variable is the force exerted (in pounds) for each hand.

The design used in the example is called a block design because the results from each person form a block. Specifically, this block design is called a matched pairs (block) design because each person provides two data observations that can be paired together (i.e. left and right hands of the same person form the pairs). Consequently, we can say that we have two dependent samples. Table 2.4 shows how a spreadsheet for the data in the matched pairs design might look.

Table 2.4. Spreadsheet of Matched Pairs (Block) Design for Example 2.6
Person Force from Right Hand Force from Left Hand

In Table 2.4, one sees that the results from each person form a block. The reason that this design is used is so that unwanted or extraneous variation can be removed from the data. In order to accomplish this goal, the data analysis is based on the differences rather than on the original data. By using the differences, we are comparing the two data observations each person provides to each other which distinguishes matched pairs from independent samples. Table 2.5 shows some data that could have been collected in this study.

Table 2.5. Picture Data of Matched Pairs (Block) Design for Example 2.6
Person Force from Right Hand (pounds) Force from Left Hand (pounds) Difference = (Right Hand Force) - (Left Hand Force)
1 47 38 47-38 = 9 pounds
2 20 15 20-15 = 5 pounds
3 33 26 33-26 = 7 pounds
" " " "
10 28 27 28-27 = 1 pound

As you examine the results from Table 2.5, you should see that there are innate differences in strength when comparing the people who participated in the study. For example, Person 1 is much stronger than Person 2. However, the variation from person to person is no longer a factor when the differences are used in the data analysis rather than using the original data.

Also, as you examine Table 2.5, you should see why we classify the two measurements for each experimental unit as the dependent. A higher value in one hand is usually followed by a higher value on the other hand. The values are more similar for each pair of measurements for each experimental unit than the values are between experimental units.

Even though the matched paired design is critical in this example, this study would also benefit from randomization. Since each person is doing both things or providing two measurements, the randomization could be used to determine the order in which the treatments are done. Why would this enhance the study? Problems can exist with block designs, including matched pair designs, when what happens with the first measurement "carries over" to the second measurement. This "carryover" effect is a type of confounding that is found with block designs.

For example, "carryover" effect could possibly occur if complicated equipment was used to measure the force exerted by a hand. If everyone used their right hand first, they might not do so well with the right hand because of not understanding the equipment, but do much better with their left hand because they learned how the equipment worked. In statistics, this is called a training effect. The opposite, however, could also take place. Suppose everyone was asked to first exert force with their right hand for 15 minutes and then repeat this task with their left hand. Participants might do okay with their right hand but become either bored or fatigued or sore when asked to repeat with this task with the left hand. So again, what happened with the first measurement would "carryover" and affect the second measurement. One may conclude that one hand is stronger than the other, not because this is really true, but because the "carryover" effect allowed this to happen.

The overall conclusion is that if you randomly assign the order of treatment, some people will use their right hand first and other people will use their left hand first. This randomization should cancel out the possibility of a "carryover" effect. In statistics, we call this a randomized block design, as shown in Table 2.6. Randomizing the order of treatment makes this a randomized experiment.

Table 2.6. Randomized Matched Pairs (Block) Design for Example 2.6
Person Hand Used First Hand Used Second
1 Right Hand Left Hand
2 Left Hand Right Hand
3 Right Hand Left Hand
10 Left Hand Right Hand

Example 2.7: Observational Study (Two Dependent Samples or Matched Pairs) Section

An owner of a theater wants to determine if the time of the showing affects attendance at a "scary" movie. In order to check this claim, a sample of five nights from all possible nights over the past month was obtained. The attendance (total number of tickets sold) for both the 7:00 PM and the 9:30 PM showings was determined for each of the five nights.

In this example:

  • the experimental unit is the night
  • the explanatory variable is the showing time
  • the response variable is the attendance at each showing.

This example also uses a matched pair (block) design because there are two measurements made on each night. A picture of this matched pair block design is found in Table 2.4.

Table 2.7. Matched Paired (Block) Design for Example 2.7
Night Attendance at 7:00 PM Showing Attendance at 9:30 PM Showing

Again, why is the matched pairs design preferred over two independent samples? In this example, our goal is to determine whether or not the time of showing affects attendance at the "scary" movie. We do not want any extraneous or other unwanted variation to explain the differences in attendance. In this example, the potential unwanted variation would be the variation that would exist from night tonight. Some of the selected nights may fall on a weekend while other nights may fall on a weekday. This factor could affect attendance. However, this will no longer be a problem when both measurements are made on the same night.

This example, however, cannot be a randomized experiment because it would be impossible to randomly assign the time of showing. The 7:00 PM show will always take place before the 9:30 PM showing. Consequently, there is a possibility that what happens at the 7:00 PM showing may "carryover" and affect attendance at 9:30 PM. A possible "carryover" effect could be the fact there is a limited amount of parking near the theatre. If this were true, perhaps those at the 7:00 PM showing take all the available spots. Then people planning to attend that 9:30 PM showing may not attend because of not being able to find a parking spot. However, this problem may not exist if there is sufficient time between the two showings so that those who attended the 7:00 PM showing had time to leave before those who arrived for the 9:30 PM showing. In any event, because this is an observational study, confounding variables are possible. "Cause and effect" conclusions may not be used if statistical significance is found.

Summary of resesarch studies Section

Table 2.8. Summary of the Four Examples
Examples Type of Study Type of Samples Randomization Used Is Confounding Possible?
2.4 Experiment Two Independent Randomize type of treatment No, randomization cancels out confounding
2.5 Observational Two Independent None Yes
2.6 Experiment Two Dependent (Matched Pairs) Randomize order of treatment No, randomization cancels out confounding
2.7 Observational Two Dependent (Matched Pairs) None Yes