Unit 49: Surveys and Studies

You will almost certainly have a survey or study type question on your test.  It will range from easy to extremely annoying.

An example:

In order to determine if treatment X is successful in improving eyesight, a research study was conducted. From a large population of people with poor eyesight, 300 participants were selected at random. Half of the participants were randomly assigned to receive treatment X, and the other half did not receive treatment X. The resulting data showed that participants who received treatment X had significantly improved eyesight as compared to those who did not receive treatment X. Based on the design and results of the study, which of the following is an appropriate conclusion? 

A)  Treatment X will cause a substantial improvement in eyesight 

B)  Treatment X improves eyesight better than all other available treatments. 

C)  Treatment X will improve the eyesight of anyone who takes it. 

D)  Treatment X is likely to improve the eyesight of people who have poor eyesight. 


  • Run through a mental checklist of the following:
    • Is the sample size large enough?  
      • Flawed:  5 people in a city were surveyed
      • OK:  3,000 people in a city were surveyed
    • Is the sample completely random?  
      • Flawed:  Researchers went to McDonald’s and asked people about a soda tax
      • OK:  Researchers mailed surveys to random people in a city to ask about a soda tax
    • Is there necessarily causation based on the study? 
      • Flawed:  a diet drug was given to overweight people who also started exercising 6 times a week and they lost weight
      • OK:  a diet drug was given to overweight people who maintained their previous diet and exercise levels and they lost weight
    • How broadly can the results be interpreted? 
      • Flawed:  a group larger than was sampled.  40 3rd graders were sampled at school, apply the results to all students at the school.
      • OK:  results within the group that was sampled.  Sample was from all elementary school students, apply to all students at that particular elementary school.

Back to our example:

As always, we have to pay very close attention to the wording.  It is important to differentiate what is likely true from a survey or study, and what must be true.

Answer A) is incorrect because it doesn’t limit improvement to only people with poor eyesight to start with.

Answer B) is incorrect because only one treatment was involved in the study.

Answer C) is incorrect because similar, to A), only people with already bad eyesight were in the study.

This leaves us with D) as the correct answer.

Margin of error

Surveys or studies may contain a margin of error.  This simply says, because we only took a sample, we don’t know the exact true value, but we know it should fall within this range.

A park ranger asked a random sample of visitors how far they had hiked during their visit.  Based on their responses, the estimated mean was found to be 4.5 miles, with an associated margin of error of 0.5 miles.  Which of the following is the best conclusion from this data?

A) It is likely that all visitors hiked between 4 and 4.5 miles

B) It is likely that most visitors hiked exactly 4.5 miles

C) It is not possible than any visitor hiked less than 4 miles

D) It is plausible that the mean distance hiked for all visitors is between 4 and 5 miles

With margin of error, the “real” average falls somewhere between 4 and 5 miles (4.5 ± .5).  Remember, though, this is the average.  If there were only 2 hikers, and one hiked 1 mile and the other 8 miles, our average would be 4.5.  This is why answers A, B, and C are incorrect.