Every significance test ends in a decision, and a decision made from random data can be wrong in two different ways. Knowing their chances helps you design a study that can actually find what you are looking for.
Key idea
Type I error: rejecting $H_0$ when $H_0$ is true (a false alarm). Its probability is $\alpha$. Type II error: failing to reject $H_0$ when $H_a$ is true (a miss). Its probability is $\beta$. Power $= 1 - \beta$ is the probability of correctly rejecting a false $H_0$. Power grows with a larger sample, a larger $\alpha$, and a true value farther from $H_0$.
Why it works
We reject when the statistic lands beyond a cutoff. If $H_0$ is true, the chance of landing there is exactly the $\alpha$ we chose. If instead the truth is some other value, the statistic's distribution is centered there, and power is the area of that distribution beyond the same cutoff. A larger sample narrows both distributions, so they overlap less; a bigger real difference pulls them apart; a larger $\alpha$ moves the cutoff toward $H_0$. Each increases power, and the last one does it by allowing more Type I errors.
| $H_0$ is true | $H_a$ is true | |
|---|---|---|
| Reject $H_0$ | Type I error (probability α) | Correct (probability = power) |
| Fail to reject $H_0$ | Correct | Type II error (probability β) |
Worked example: Naming the errors
A smoke detector "tests" $H_0$: there is no fire. Describe a Type I and a Type II error, and which is worse.
- Type I: rejecting "no fire" when there is no fire: the alarm sounds for burnt toast.
- Type II: failing to reject "no fire" when there is a fire: the alarm stays silent.
- A Type II error is far worse here, so detectors are designed to be sensitive (high power), accepting more false alarms.
Answer: False alarm versus a missed fire; the miss is worse.
Worked example: Computing power
Test $H_0: \mu = 100$ against $H_a: \mu > 100$ with $\sigma = 10$, $n = 25$ and $\alpha = 0.05$. Find the power if the true mean is 104. ($P(Z < 0.36) = 0.6406$.)
- SD of $\bar{x}$: $10/\sqrt{25} = 2$.
- Cutoff from $H_0$: reject when $\bar{x} > 100 + 1.645(2) = 103.29$.
- If $\mu = 104$: $$P(\bar{x} > 103.29) = P\left(Z > \dfrac{103.29 - 104}{2}\right) = P(Z > -0.36)$$
- By symmetry that is $P(Z < 0.36) = 0.6406$.
- So $\beta \approx 0.36$: the test misses a real difference of 4 more than a third of the time.
Answer: Power ≈ 0.64.
Worked example: Raising power
Repeat with $n = 100$. ($P(Z < 2.36) = 0.9909$.)
- SD of $\bar{x}$: $10/\sqrt{100} = 1$; cutoff $100 + 1.645 = 101.645$.
- $z = \dfrac{101.645 - 104}{1} \approx -2.36$, so power $= P(Z > -2.36) = 0.9909$.
Answer: Power ≈ 0.99: four times the sample, far fewer misses.
Common mistake
Mixing up the two errors. Type I is the one that rejects a true null hypothesis (convicting an innocent defendant); Type II keeps a false one (letting a guilty one go). Check by writing $H_0$ in words and asking: did we reject it, and was it actually true?