A random sampling method aims at the truth. A biased method aims somewhere else, and it misses in the same direction every time you use it. This lesson names the usual ways surveys go wrong.
Key idea
Bias is a systematic error from the way data are collected. Common sources: undercoverage (some groups cannot be chosen), nonresponse (chosen people do not answer), voluntary response (people choose themselves, usually those with strong opinions), convenience sampling, and response bias (leading questions, sensitive topics, interviewer effects). A bigger sample does not reduce bias.
Why it works
Picture throwing darts. Variability is how scattered the darts are; bias is how far the center of the cluster is from the bull's-eye. Taking a larger random sample tightens the cluster, but if the method aims at the wrong spot, a tight cluster in the wrong spot is still wrong, and now it looks convincing. Only a better method moves the aim.
Worked example: Naming the bias
A city council wants to know whether residents support a new dog park. It posts a link on the city's pet-adoption page asking visitors to vote. Of 900 votes, 88% say yes. What is the problem?
- People chose whether to respond: voluntary response.
- The link was on a pet page, so dog owners were far more likely to see it: undercoverage of everyone else.
- Both push the result toward "yes," so 88% probably overstates support.
Answer: Voluntary response and undercoverage; the estimate is likely too high.
Worked example: Question wording
Compare: "Do you support the new library?" with "Do you support raising your taxes to build a new library?" Why might the results differ?
- The second version reminds people of a cost, the first does not.
- Changing the wording changes the answers people give: response bias.
- A fair survey uses neutral wording and the same question for everyone.
Answer: The wording itself changes the responses.
Worked example: Size does not fix bias
In 1936 a magazine mailed millions of ballots, mostly to people listed as car owners and telephone subscribers, and about 2.4 million came back predicting a clear win for Alf Landon. Franklin Roosevelt won in a landslide. Why did such a huge sample fail?
- In 1936 car and telephone owners were wealthier than voters overall: undercoverage.
- Only a fraction of those mailed answered: nonresponse.
- Millions of biased answers give a very precise estimate of the wrong number.
Answer: The method was biased; size could not save it.
Common mistake
Believing that a larger sample fixes a biased sampling method. More responses collected the same way repeat the same bias. Check the method first: who could be chosen, who answered, and how the question was worded.