Overview Section
Suppose you desire to do a study or administer a survey. As an investigator, the most challenging task that you will confront is to decide what questions to ask and/or what measurements to obtain. In this lesson, you will be introduced to some key concepts associated with obtaining measurements. You will also learn about possible pitfalls found with survey questions.
It's All in the Wording
There are many possible pitfalls that can lead to bias when asking questions in a survey or study. In thinking about the veracity of results from a sample survey ask yourself if any of these pitfalls are likely to be a problem. It is also possible that more than one type of pitfall can happen at the same time. Examine the following examples.
Example 1.4 Deliberate Bias such as One-Sided Statements Section
People who use a form of deliberate bias often desire to gather support for a specific cause or opinion. Consider two different wordings for a particular question:
Wording 1: It is hard for today's college graduates to have a bright future with the way things are today in the world.
- agree
- disagree
Wording 2: Today's college graduates will have a bright future.
- agree
- disagree
Although Wording 1 and Wording 2 are contradictory statements, when both questions are used in the same survey, it is not uncommon to find that people answer "agree" to both questions. This is because respondents tend to agree to one-sided statements. Listed below are revised wordings for these two questions. These choices are preferred because the statements are now at least two-sided.
Revised Wording 1: Do you agree or disagree that it is hard for today's college graduates to have a bright future with the way things are today in the world?
Revised Wording 2: Do you agree or disagree that today's college graduates will have a bright future?
Example 1.5. Filtering Section
Consider two different choices of answers for a particular question:
Choice 1: What is your opinion of our current President?
- favorable
- unfavorable
Choice 2: What is your opinion of our current President?
- favorable
- unfavorable
- undecided
This example illustrates the problem of "filtering." Filtering exists when certain choices such as "undecided" or "don't know" are not included in the list of possible answers. People tend to provide an answer of "undecided" or "don't know" only when these choices are included in the list of possible answers.
Example 1.6. Importance of Order Section
Consider two different wordings for a particular question:
Wording 1: Pick a color: red or blue?
Wording 2: Pick a color: blue or red?
The results in Table 1.1 are from a study conducted in a Statistics class. As you can see the results vary somewhat based on the order in which the colors are presented. Even though many people probably have a preference for one color over the other, if order does not matter, the percents should be same with each wording.
Table 1.1. Bias due to Order of Comparisons
Color Choice | Wording 1 | Wording 2 |
Red | 59% | 45% |
Blue | 41% | 55% |
Example 1.7. Anchoring Section
Consider two different wordings for a particular question:
Wording 1: Knowing that the population of the U.S. is 316 million, what is the population of Canada?
Wording 2: Knowing that the population of Australia is 23 million, what is the population of Canada?
This survey was conducted in Stat 100 classes where both wordings of the question were randomly distributed. The students did not know that there were two versions of this question so each only answered the question that they received. The results of this survey are found in Figure 1.4.
Figure 1.4. STAT 100 Survey Results
As you can see, the students were influenced by the wording of the question that they were asked to answer. People's perceptions can be severely distorted when they are provided with a reference point or an anchor. People tend to say close to the anchor because of either having limited knowledge about the topic or being distracted by the anchor. You should also consider the following three points:
- The sample sizes were large enough to detect a difference between the two groups
- Canada's population is about 35 million
- The anchor might be less distracting if the following wording were used: "What is the population of Canada when knowing that the population of the U.S. is 316 million?" but it is best to leave out the anchoring statement altogether.
Example 1.8. Unintentional Bias Section
Consider two different wordings for a particular question:
Wording 1: Do you favor or oppose an ordinance that forbids surveillance cameras to be placed on Beaver Avenue?
Wording 2: Do you favor or oppose an ordinance that does not allow surveillance cameras to be placed on Beaver Avenue?
People will tend to answer "oppose" or "no" to a question that contains words such as forbid, control, ban, outlaw, and restraint regardless of what question is actually being asked. People do not like to be told that they can't do something. So the responses to the two questions would not provide similar results. Wording 2 would be preferred over Wording 1.
Example 1.9. Unnecessary Complexity ("Double-Barreled" Problem) Section
Consider the following question.
Question: Do you think that health care workers and military personnel should be the first to receive the smallpox vaccination?
The problem with this question is that the respondent must consider both health care workers and military personnel at the same time. The following rewording is much better.
Revised Question: Who should have priority in receiving the smallpox vaccination?
- health care workers
- military personnel
- both health care workers and military personnel
- neither
Example 1.10. Asking the Uninformed and Unnecessary Complexity (Double Negative Problem and List Problem) Section
Consider the following question:
Question: Do you agree or disagree that children who have a Body Mass Index (BMI) at or above the 95th percentile should not be allowed to spend a lot of time watching television, playing computer games, and listening to music?
The first concern with this question is that many people may not clearly understand what the Body Mass Index (BMI) represents. BMI is a measure that is used to identify obesity and is calculated by dividing a person's weight (in kilograms) by the square of their height (in meters). (Note: many Web sites have BMI calculators.) In children and adolescents, obesity is defined as a BMI for age and gender at or above the 95th percentile. This definition should be included prior to the listing of the question on a survey.
This question can also cause problems because of a possible "double negative". Specifically, the problem is with the "disagree" choice. This choice produces a double negative because "disagree" and "should not" are both in the statement. Many respondents will not understand what they are really saying. (It is easy to make the mistake of the double negative).
Revised Question-First Revision: Do you agree or disagree that children who have a Body Mass Index (BMI) at or above the 95th percentile should spend less time watching television, playing computer games, and listening to music?
As you examine this revised question you should also note that there still is a list of three choices embedded in the questions. Respondents sometimes can get hung up on the list of choices (see the "double-barreled" problem above). For example, they may feel that watching television is a bad idea for obese children but listening to music is not.