A coin lands heads 64 times in 100 flips. Is it unfair, or could a fair coin do that by chance? You answered questions like this with simulations. A significance test does the same thing with the sampling distribution in place of the simulation.
Key idea
A significance test starts with a null hypothesis $H_0$ (no effect, no difference; a specific parameter value) and an alternative $H_a$ (what we suspect). The p-value is the probability, computed assuming $H_0$ is true, of getting a result at least as extreme as the one observed. If the p-value is below the significance level $\alpha$ (often $0.05$), we reject $H_0$: the data give convincing evidence for $H_a$. Otherwise we fail to reject $H_0$.
Why it works
The logic is like a proof by contradiction, with chance in it. Suppose $H_0$ is true. Then we know exactly how the statistic behaves and can ask how surprising our data are. A tiny p-value means: "if $H_0$ were true, we would almost never see data like these." Either something very unusual happened, or $H_0$ is wrong; we take the second option. A large p-value means the data are the kind of thing $H_0$ produces routinely, so they give no reason to doubt it. Like a court verdict of "not guilty," failing to reject is not a proof of innocence.
Worked example: A coin
A coin lands heads 64 times in 100 flips. Test $H_0: p = 0.5$ against $H_a: p \ne 0.5$. ($P(Z < 2.80) = 0.9974$.)
- If $H_0$ is true, $\hat{p}$ is about normal with mean 0.5 and SD $\sqrt{0.5 \cdot 0.5/100} = 0.05$.
- $z = \dfrac{0.64 - 0.5}{0.05} = 2.8$.
- Two-sided: count results this extreme in either direction. $$P(Z > 2.8) = 1 - 0.9974 = 0.0026$$ so the p-value is $2 \times 0.0026 = 0.0052$.
- $0.0052 < 0.05$: reject $H_0$. A fair coin would almost never give a result this lopsided.
Answer: p-value $\approx 0.005$; convincing evidence the coin is unfair.
Worked example: Reading a p-value
A study of a new tutoring program reports a p-value of $0.03$ for $H_0$: the program has no effect. Which reading is correct?
- Correct: if the program truly had no effect, results at least this large would happen only about 3% of the time.
- Not correct: "there is a 3% chance the program has no effect." The p-value was computed by assuming no effect; it cannot also be the probability of that assumption.
Answer: It is a probability about the data, given $H_0$.
Worked example: A large p-value
Another test gives a p-value of $0.08$ at $\alpha = 0.05$. What should you conclude?
- $0.08 > 0.05$: fail to reject $H_0$.
- This does not show $H_0$ is true. The data are simply not strong enough to rule it out; a larger study might.
Answer: Fail to reject $H_0$; not convincing evidence.
Common mistake
Reading the p-value as the probability that the null hypothesis is true. The p-value is $P(\text{data this extreme} \mid H_0)$, and that is a different thing from $P(H_0 \mid \text{data})$, just as in Bayes' rule a test's accuracy differs from the chance that a positive result is right. Check your sentence: it should begin "If $H_0$ were true, …".
"Statistically significant" means "hard to explain by chance," not "large" or "important." With a huge sample, a tiny, unimportant difference can have a very small p-value. Always report the size of the effect too.