Hundreds of research studies are published each week and many of these find their way to become the basis of reports in the popular press or on websites devoted to providing science news to the public. In Lessons 1 and Lesson 2 we have seen how important it is to critically evaluate the process used to gather the data of the study as we decide on the veracity of the claims made.

In this lesson, we review this material and provide step-by-step guidelines for evaluating research studies like examples 3.1, 3.2, and 3.3.

Evaluating Research Studies

1. Step 1

   Determine the type of study conducted (e.g. sample survey, observational study, randomized experiment). Example 3.1 is based on data from a sample survey; example 3.2 is based on data from a prospective observational study, and example 3.3 is based on data from a randomized experiment.

2. Step 2

   Determine the critical components of the research (See pages 20-23 in the text). Here you must seek answers to questions like who funded the research? Who were the subjects and how did they come to take part? What are the exact measurements made or questions asked? What was the research setting? What differences between comparison groups might exist besides the factor of interest? How big of an effect was seen and did the researchers claim it was statistically significant (unlikely to happen by chance)?

   Step two lies at the heart of the material in Lessons 2 and 3. Have a look at the three examples above and judge for yourself whether understanding the critical components suggests potential sources of bias.

3. Step 3

   Check the "Difficulties and Disasters" that we saw in earlier lessons.

   1. For sample surveys:
      • check if a probability method was used in generating the sample
      • check if the sampling frame was close to the population of interest
      • check if there were difficulties in reaching certain parts of the sampling frame that could be related to what's being studied
      • check the response rate and think about how non-respondents might differ from respondents
   2. In example 3.1, there were two surveys - one of the public and one of the scientists. The "public" sample involved 2002 people chosen randomly using a stratified sample from a sampling frame that included American adults with access to a landline or cell phone. Interviews were conducted in either English or Spanish depending on the preference of the respondent. Because different demographic groups have very different response rates, the Pew Center weights the responses in a way to bring them into alignment with census data on the true proportions of the population for factors like age, gender, race, and education level. The exact response rates are not provided for this survey but are given in great detail by Pew for many studies. Typically they find that there is no answer for about 1/3 of the working phone numbers they dial and that about 60% of the people they contact refuse to participate and another 10% are ineligible (e.g. a child's phone). That leaves a response rate of about 20% of working phone numbers they call. This may seem low, but it is actually on the high end for the polling industry - that's why survey organizations spend so much effort in their methods for weighting the results given to adjust for ways that the respondents might be different from the non-respondents. The poll of scientists involved 3,748 scientists chosen randomly from the membership list of AAAS which includes about 150,000 scientists. About 19,000 members were sent e-mails to introduce the study so the response rate was around 20%. Pew again weighted the results to align the sampled scientists with the sampling frame according to membership status (graduate student member/active faculty/retired faculty, and a fellow of the society versus non-fellows). Two areas might be of concern here. First, we notice that people in the general public survey were contacted by phone while the scientists were initially contacted by e-mail. Second, the membership of AAAS (the sampling frame) may be different from the population of scientists in general (e.g. different fields are more heavily represented in AAAS than others).
   3. For comparative observational studies:
      • check for possible confounding variables and whether claims of causation were made (ask: Is there an important factor related to what brought the subjects into the groups being compared?)
      • check whether claims about generalizing the results to a larger population are appropriate
      • check whether a retrospective or prospective design was used (if retrospective - is there a reliance on the accuracy of a subject's memory?)

      In example 3.2 the data was gathered prospectively using data on prescription drug reimbursements from a large HMO in the Seattle, Washington area (note that the study was about prescription drugs even though the headlines mention over-the-counter drugs!). The prospective design and avoiding patient self-reports about drug use were strengths of this study (the accuracy of self-report data is always a concern). Generalizing the results to the public at large might be a concern if the elderly membership of this HMO differed significantly from the general elderly population in ways that might affect dementia. For example, having health insurance in the first place indicates an economic and educational status that might be associated with the onset of dementia.

   4. For randomized experimental studies:
      • check for possible confounding variables when small samples are used (with large samples the randomization will help with this)
      • check for interacting variables (ask: Do the results stay the same for different sub-populations?)
      • check for placebo, Hawthorne, and experimenter effects (ask: Was the experiment double-blind? Would subjects behave differently because they are being studied?)

      In example 3.3 the subjects could not be blinded as to whether they had their eyes closed or not, but the researchers evaluating how well they did in remembering the events they had seen were blinded. The task in this experiment involved watching a video showing an electrician stealing items from a job site. An important concern here might be the artificial nature of the setting in which the experiment was done (watching a video of a robbery compared to see a robbery in person for example). Would the results carry over to recalling important events in real life?

4. Step 4

   If the information needed to critique is not provided in the report, try to find the original source and see if the missing information is provided there.

   Each of the three examples here provides a direct link to the full original report although, for example, 3.3, the original scientific publication must be purchased. Example 3.2 is a good example to see how there can often be a good deal of difference between the information provided in the original scientific report and the information in a report in the popular press and especially in the headline used to promote the article.

5. Step 5

   Do the results make sense? Are they based on a sound scientific footing?

   For example, the link between long-term heavy cell phone use and brain cancer is very controversial. The association has been seen in a number of retrospective case-control studies but no laboratory experiments have been able to provide a biological mechanism for how a causal link might happen. Perhaps the retrospective studies are flawed because the memories of patients with brain tumors about their cell phone use years earlier differ in accuracy from the memories of the control groups studied.

6. Step 6

   Ask if there is an alternative explanation for the results?

   In example 3.1 some of the issues on which public opinion is seen to be quite different from the opinion of scientists involve public policy controversies. Since opinions on such topics are strongly associated with political beliefs - it is possible that political affiliations are a confounding variable that explains some part of the differences seen.

7. Step 7

   Decide if the results are strong enough to encourage you to change your behavior or beliefs.

   Does example 3.1 suggest to you that there is a greater need for better science education in the United States? Would example 3.2 lead you to examine the ingredients of prescription drugs you use and seek alternatives if you spotted the ingredient in the study? Would example 3.3 encourage you to ask someone to close their eyes in order for them to recall a memory you are asking them about?

Here are four different graphs that can be used to describe measurement data. These graphs include:

1. Dotplots
2. Stemplots (Stem and Leaf Plot)
3. Histograms
4. Boxplots (Covered in section 3.4)

Dotplots

The dotplot is the first graph that will be used to display this sample of 40 ages. The purpose of the dotplot is to represent each observation as a dot. On this dotplot, you will find that the ages range from 32 to 63 years. You should also notice that there is more tenured faculty at older ages.

Figure 3.2. Dotplot (Ages of 40 PSU Tenured Faculty)

Stemplots (Stem and Leaf Plot)

The second graph that is possible with measurement data is a stemplot. Stemplots concisely display the data in order from smallest to largest. Below is the list of the 40 ages in order from youngest to oldest.

These forty observations are displayed in the stemplot found in Figure 3.3. In this stemplot, we again find that the range of ages spans from 32 years to 63 years. We also find that there is more tenured PSU faculty at older ages. Stemplots can provide useful information about small data sets.

Stem and Leaf Plot: Ages of Tenured PSU Faculty
(each row represents 5 years)
Stem-and-leaf of Ages N = 40
Leaf Unit = 1.0

\(
\begin{array}{lll}
1 & 3 & 2 \\
3 & 3 & 79 \\
6 & 4 & 023 \\
12 & 4 & 567789 \\
(10) & 5 & 0112333444 \\
18 & 5 & 55667888999 \\
7 & 6 & 0011233
\end{array}
\)

Histograms

The third graph is called a histogram. Of all the graphs presented so far, the histogram may be the most valuable. A histogram is essentially a bar graph for measurement data. The difference, however, between a histogram and a bar graph, is that with a histogram the categories are a range of numbers rather than words. Usually, each numerical category must have the same width. The heights of the bars either reflect the frequency or the relative frequency (percents) of encountering that range of numbers in the data

Recall that the ages span from 32 to 63 years. The range of these ages is (63-32) = 31 years. Looking at the stemplot, one finds that there is more tenured faculty at the older ages. We want to be able to show this trend. There need to be enough categories to properly display any trend. If we only choose 4 categories (since we have tenured faculty in their 30s, 40s, 50s, and 60s) we would not be able to detect this trend as well. If we choose 9 categories the trend will become more obvious. Statistical software will usually make this determination for you.

Width of each Category = Range/ number of categories = 31/9 = 3.4 (rounded to 4 years)

The starting point is always below our lowest observed value and the ending point is always above our highest observed value. For readability, it is also best to have the intervals start and end on whole numbers (or easy to digest numbers if the data are on a different scale). In this instance, our starting point of 30 years is below the lowest observed age which is 32 years. The ending point of 66 years is above our highest observed age which is 63 years. In order to make sure that an observation only falls into one category, we construct our 4 year categories as shown in Table 3.1 below. Our first category includes ages starting at 30 years and ending just below 34 years. An observed age of 34 is not included in the first category but rather is included in the second category. Using this method, no observed age can fall into more than one category. The observed ages are then placed into the appropriate category and the histogram is constructed.

Recall:

The information from Table 3.1 is used make the histogram found in Figure 3.4. The horizontal axis displays the categories while the vertical axis displays the percent of the observations (ages) found in each category. (Note: You will not be asked to make histograms. You will only be asked to interpret them. However, it is important to see how one is made so that you understand the interpretation.)XXXXXXXXXXXXXXXXXXXX

Figure 3.4. Histogram (Ages of PSU Tenured Faculty)

As you look at the histogram, you should notice that there are more ages in the upper half of the graph. In statistics, when data on a histogram is off-center, the data is labeled as skewed. In this case, the data is skewed to the left because a larger percent of the ages are found in the upper tail.

We will discuss two important ways to summarize measurement data. These include:

1. measures of center (where the data are located along the number line)
2. measure of spread (how much variation there is about the center)

To represent the center of a list of measurement data we focus on:

1. The mean (the numerical average)
2. The median (the 50th percentile)

A percentile is the position of an observation in the dataset relative to the other observations in the data set. Specifically, the percentile represents the percentage of the sample that falls below this observation. For example, the median is also known as the
50th percentile because half of the data or 50% of the observations lie below the median. There are three percentiles that will be of interest to us.  Figure 3.7 shows these percentiles (quartiles) graphically.

Percentiles of Interest

Percentile Alternate Names Interpretation 25th percentile

• Lower Quartile (QL)
• First Quartile (Q1)

25% of the data falls below this percentile 50th percentile

• Median
• Second Quartile ( Q2)

50% of the data falls below this percentile 75th percentile

• Upper Quartile (QU)
• Third Quartile (Q3)

75% of the data falls below this percentile

Figure 3.7. Quartiles for a Distribution

A five-number summary is a useful summary of a data set that is partially based on selected percentiles. Below are the five numbers that are found in a five-number summary. 

Figure 3.8. Five-Number Summary

The five-number summary is also of value because it is the basis of the boxplot. Figure 3.9 below is a vertical boxplot of the variable salaries. The first thing to consider in this graph is the box. The ends of the box locate the lower quartile and upper quartile, which in this case are 33,500 and 49,000 dollars respectively. The line in the middle of the box is the median. As you examine the box portion of the box, you should notice that the median is closer to the lower quartile than to the upper quartile. This suggests that data set is skewed and specifically skewed to the right. In this instance the largest observation is represented with an asterisk. Since this observation is an unusually large salary of $110,000, the graph identifies this observation as an outlier or unusual observation. Appropriate statistical criterion is used to determine whether or not an observation is an outlier. Lines called 'whiskers' extend from the box out to the lowest and highest observations that are not outliers. Notice that the whisker on the bottom is much shorter than the whisker on the top of this boxplot. This is another hallmark of a distribution that is skewed to the right (because the first 25% of the data covers a narrow length on the number line while the last 25% are more spread out.

Figure 3.9. Horizontal Boxplot of Salaries

One of the most important uses of the boxplot is to compare two or more samples of one measurement variable.

Two ways to represent the spread or variation are:

1. Interquartile Range (IQR)
2. Standard Deviation (SD)