News stories, ads and social media posts are full of statistics. You now know enough about sampling and experiments to ask the right questions before believing a claim.
Key idea
Judge a statistical claim by how the data were produced. Ask: Who was studied, and how were they chosen? Was anything randomly assigned? How many were studied, and how much did results vary? What exactly was measured or asked? Does the conclusion match the design (association versus cause, these subjects versus everyone)?
Why it works
A number is only as good as the process that made it. A precise-looking "73%" from a self-selected online poll carries less information than a rough "about 60%" from a careful random sample. Most misleading claims are not false arithmetic; they are true numbers attached to the wrong conclusion.
- Sample: random, or convenience or volunteers?
- Design: experiment or observational study?
- Size and variation: could chance explain the result?
- Measurement: neutral wording? What was actually counted?
- Conclusion: does "causes" or "all Americans" go beyond the design?
Worked example: A causal headline
A headline reads: "Chess makes kids smarter: chess club members get higher grades." The story compares club members with other students. Evaluate.
- Students chose to join the club: an observational study.
- Students who join a chess club may already like school and get support at home (confounding).
- The data show an association; they do not show that chess causes better grades.
Answer: The headline claims cause from an observational comparison.
Worked example: Relative versus absolute risk
An ad says a medicine "cuts the risk of a side effect in half," from 2 in 10,000 people to 1 in 10,000. Evaluate.
- Relative change: from 2 to 1 is indeed half.
- Absolute change: 1 fewer case per 10,000 people, or 0.01%.
- Both are true, but "in half" sounds far bigger than the actual difference.
Answer: A large relative change can be a tiny absolute change; report both.
Common mistake
Trusting a result because the number looks precise or the sample is large. A poll of 50,000 self-selected website visitors is less trustworthy than a random sample of 1,000. Check the design questions first, then the numbers.
Misleading graphs are a cousin of misleading claims: a bar graph whose axis starts at 950 instead of 0 can make a 2% change look like doubling. Read the axis labels.