Econometrics · Chapter 3
Estimation and Inference
You never see the whole population, only a sample, so every estimate comes with a question: how far off might it be? Here is a machine that answers it. It draws 100 samples, builds 100 confidence intervals, and shows you which ones caught the truth. Then throw darts with four different estimators, and run hypothesis tests where you can see the p-value as an area, the two ways of being wrong, and the power to find what you’re looking for.
The population of commute times
Each sample and its interval
Live numbers
Missions
What you’re seeing
The population is every commute in a city, and in this simulation we know its true mean: μ = 30 minutes, the gold line. In real life you would never know it. That is the point of estimating.
Each horizontal line comes from one random sample of n commutes. The dot is the sample’s average; the line around it is a confidence interval built from that sample alone. Green lines caught the true mean; red ones missed. The intervals jump around because the samples do.
The gauge at the bottom keeps a running tally across every batch. At 95% confidence, about 95 intervals in 100 catch the truth, in the long run. Which 95? You can’t tell from any one interval. That is what “95% confidence” means: a promise about the method, not about one interval.
The methods. “t with s” is the correct recipe. “z with s” uses the normal’s 1.96 even though σ was estimated, and comes out too narrow in small samples. “z with known σ” cheats by using the true σ, which you rarely have.
Estimators and what makes them good
An estimator is a recipe that turns a sample into a guess about a population number, called a parameter. The sample average Ȳ = (Y₁ + … + Yn)/n is an estimator of the population mean μ; the number it gives for your data is an estimate. Because the sample is random, the estimator is a random variable with its own distribution, the sampling distribution. We judge estimators by that distribution:
Right on average: E(θ̂) = θ. Over many samples the estimates center on the truth. The sample mean is unbiased: E(Ȳ) = μ.
Closes in on the truth as the sample grows: for any margin, the chance θ̂ misses by more than it goes to zero. Var(Ȳ) = σ²/n shrinks to zero, so Ȳ is consistent.
Among unbiased estimators, has the smallest variance. Using all n observations beats using one: Var(Ȳ) = σ²/n versus σ² for Y₁ alone.
Bias and variance combine into one measure of how far off an estimator tends to be, the mean squared error:
Throw
Goals
The four estimators all try to hit the center of a population (two numbers at once, so it fits a dartboard; the population’s spread is 1 in each direction). The sample mean uses everyone. First person only ignores all but one person: still unbiased, never better as n grows. Shrunk toward a guess blends the sample mean with a fixed guess: λȲ + (1 − λ) × guess. It is biased unless the guess is right, but less variable. Volunteers only averages people who chose to answer, and people with a larger first number are more likely to volunteer.
What to notice. Raise n and the sample mean’s darts close in on the bullseye; the first-person darts don’t budge. The volunteers’ darts tighten too, around the wrong spot: a bigger biased sample is just a more confident wrong answer. And with tiny samples, the shrunk estimator can beat the sample mean on mean squared error. That bias-for-variance trade is the idea behind many modern methods (Chapter 17).
Take samples of n = 4 from the dartboard’s population. The sample mean has no bias and variance 1/4 in each direction, so its MSE (adding both directions) is 2 × 0.25 = 0.5, a typical miss of √0.5 = 0.71.
Shrink halfway to a guess that is 0.71 away from the truth. The variance falls to 0.5² × 0.5 = 0.125, and the bias is 0.5 × 0.71 = 0.355, so MSE = 0.125 + 0.355² = 0.25, a typical miss of 0.50. Better, despite the bias. With n = 25 the sample mean’s MSE is only 0.08 and the shrunk estimator’s bias of 0.355 alone loses: its MSE is at least 0.126. The simulation agrees: root mean squared errors of 0.71 versus 0.50 at n = 4, and 0.28 versus 0.38 at n = 25.
Standard errors and confidence intervals
The spread of an estimator’s sampling distribution is its standard error. For the sample mean it is σ/√n. We don’t know σ, so we estimate it with the sample standard deviation s, which divides by n − 1 rather than n (one degree of freedom goes into estimating the mean; with n − 1, s² is unbiased for σ²):
If the data are normal, (Ȳ − μ)/se(Ȳ) has a t distribution with n − 1 degrees of freedom, so an interval that catches μ in 95% of samples is
For other levels, use other percentiles: a 99% interval uses the 99.5th, so it is wider. More confidence costs width; more data buys narrowness, but only at the rate √n.
A coffee chain times 25 drive-through orders at one store: average 3.8 minutes, standard deviation 1.5. The standard error is 1.5/√25 = 0.30. With 24 degrees of freedom, t* = 2.064, so the 95% interval is 3.8 ± 2.064 × 0.30 = 3.8 ± 0.62, or 3.18 to 4.42 minutes. To cut the margin to ±0.2 minutes, you’d need about (2 × 1.5/0.2)² ≈ 225 orders.
Hypothesis tests and p-values
A hypothesis test checks a claim about a parameter against the data. The null hypothesis H₀ is the claim being tested, often “no effect” (μ = μ₀); the alternative H₁ is what you conclude if the data are inconsistent with it (μ ≠ μ₀, or one-sided, μ > μ₀). The test statistic measures how far the estimate is from μ₀ in standard errors:
The p-value is the probability, if H₀ were true, of a test statistic at least as extreme as the one observed. Small p-values mean the data would be surprising if the null were true. Reject when p < α.
The test
The world and the study
Goals
Two ways to be wrong
| H₀ is true | H₀ is false | |
|---|---|---|
| Reject H₀ | Type I error (false alarm). Probability α, which you choose. | Correct. Probability 1 − β, the power. |
| Don’t reject H₀ | Correct. | Type II error (a miss). Probability β. |
Lowering α makes false alarms rarer and misses more common. Power rises with the size of the true effect and with the sample size, and falls with noise. For a two-sided test of a mean with known σ, an effect of d standard deviations has power of about Φ(d√n − 1.96) at α = 5% (plus a tiny chance of rejecting in the wrong direction). For 80% power you need d√n ≈ 1.96 + 0.84 = 2.80.
H₀: μ = 3.5 against H₁: μ ≠ 3.5. With Ȳ = 3.8 and se = 0.30, t = (3.8 − 3.5)/0.30 = 1.00. With 24 degrees of freedom the two-sided p-value is 0.33. Data like these would be common if the store really averaged 3.5 minutes, so we don’t reject. Notice that 3.5 lies inside the 95% interval of 3.18 to 4.42: a two-sided test at 5% rejects exactly when μ₀ falls outside the 95% interval.
A manager wants to detect a change of 0.3 standard deviations in average order time with 80% power, at α = 5%, two-sided. Solve d√n = 2.80: √n = 2.80/0.3 = 9.34, so n ≈ 87.2; round up to 88. To detect half that effect, 0.15 sd, takes four times the data: about 349.
A p-value is computed assuming H₀ is true, so it can’t be the probability that H₀ is true. It is the chance of data at least this extreme if the null held. It also says nothing about how big or important an effect is, and a p-value above 0.05 is not evidence that there is no effect, only that the data can’t rule out zero. In 2016 the American Statistical Association issued a statement making these points, and warned against reducing research to “p < 0.05” or not.
Statistical versus practical significance
Statistically significant means the data are hard to square with no effect. Practically significant means the effect is big enough to matter. With enough data, a tiny effect becomes statistically significant: a website with 10 million visitors can detect a change in click rates of a few hundredths of a percentage point. Whether that change is worth anything is a business question, not a statistical one. Always report the estimate and its confidence interval, not just whether p < 0.05.
Precision is not validity. Before launching New Coke in 1985, Coca-Cola ran nearly 200,000 blind taste tests. With that many people, the standard error of the share preferring the new taste was tiny. But a sip of an unlabeled cola measured the wrong thing: people buy a brand and drink whole cans. A huge sample estimated the wrong quantity very precisely (the case). No standard error can fix a question that doesn’t match the decision.
A short history, and the people
R. A. Fisher popularized significance tests and p-values in Statistical Methods for Research Workers (1925), written for scientists at an agricultural research station. He treated the p-value as a graded measure of evidence and suggested 1 in 20 as a convenient line, which hardened into the 5% convention.
Jerzy Neyman and Egon Pearson framed testing as a decision between two hypotheses in 1933, introducing the two kinds of error and the power of a test. Neyman added confidence intervals in 1937. Fisher and Neyman disagreed sharply about what these tools meant; modern practice blends their ideas.
William Sealy Gosset, writing as “Student” in 1908, worked out the t distribution so that brewers could draw conclusions from a handful of measurements. Fisher later showed how to use it for regression coefficients, which is how you’ll meet it in Chapter 5.
Where you see it
Quality control. A factory that stops the line whenever a measurement looks off trades false alarms (stopping for nothing) against misses (shipping defects). Toyota’s andon cord lets any worker stop the line, accepting more false alarms to catch problems early (the case).
A/B tests. Online businesses test two versions of a page at random and use exactly these tests. Running many tests and reporting only the significant ones inflates false alarms: with α = 5%, one in twenty tests of changes that do nothing will look like a win.
Courts. “Innocent until proven guilty” makes innocence the null hypothesis. Convicting an innocent person is a type I error; acquitting a guilty one is a type II error. A higher bar for proof means fewer of the first and more of the second.
Polls and government data. Every margin of error is a confidence interval. The Bureau of Labor Statistics, for example, publishes how large a monthly change in employment must be to be statistically significant.
Practice
Standard error. A sample of 64 households spends an average of $180 a week on groceries, with a standard deviation of $48. What is the standard error of the mean?
Show the answer
se = 48/√64 = 48/8 = $6.
An interval. Build a 95% confidence interval for the households’ average weekly grocery spending (use t* ≈ 2.00 for 63 degrees of freedom). What would change for a 99% interval?
Show the answer
180 ± 2.00 × 6 = 180 ± 12, or $168 to $192. A 99% interval uses t* ≈ 2.66, giving 180 ± 16: wider, because more confidence costs width.
What does it mean? A friend says: “There is a 95% probability that the true average is between $168 and $192.” What is the careful way to say it?
Show the answer
The true average is a fixed number; this particular interval either contains it or doesn’t. The 95% describes the method: intervals built this way catch the true mean in 95% of samples. The machine at the top shows exactly that.
A test. A cereal maker claims boxes hold 500 g. In 36 boxes the average is 497.6 g with s = 6 g. Test the claim at 5%, two-sided.
Show the answer
se = 6/√36 = 1, so t = (497.6 − 500)/1 = −2.4. With 35 degrees of freedom the two-sided p-value is 0.022 < 0.05: reject the claim. The 95% interval, 497.6 ± 2.03, runs from 495.6 to 499.6 and excludes 500, as it must.
Bias and consistency. To estimate the mean, someone proposes (Y₁ + … + Yn)/(n + 1). Is it unbiased? Consistent?
Show the answer
Its expected value is nμ/(n + 1), so it is biased toward zero by μ/(n + 1). But the bias and the variance (nσ²/(n + 1)²) both shrink to zero as n grows, so it is consistent. Biased in small samples, right in large ones.
Sample size for a poll. How many people must a poll survey for a 95% margin of error of ±2 percentage points, if the true share is near 50%?
Show the answer
Margin = 1.96 × √(0.5 × 0.5/n) = 0.02, so √n = 1.96 × 0.5/0.02 = 49, and n = 2,401. Compare ±3 points with about 1,067: shrinking the margin by a third more than doubles the cost.
Power. A study has 80% power to detect an effect of 0.4 standard deviations at α = 5%, two-sided. About how much power does it have to detect an effect of 0.2? What sample size would restore 80%?
Show the answer
80% power means d√n ≈ 2.80, so √n = 7.0 and n ≈ 49. For d = 0.2, d√n = 1.4, and power ≈ Φ(1.4 − 1.96) = Φ(−0.56) ≈ 29%. To get back to 80% you need √n = 14, so n ≈ 196: half the effect takes four times the data.
Many tests. A marketing team tests 20 new ad designs against the old one, each at α = 5%. None of the new designs is any better. What is the chance at least one looks significantly better or worse anyway (treat the tests as independent)?
Show the answer
Each test avoids a false alarm with probability 0.95, so all 20 do with probability 0.95²⁰ = 0.358. The chance of at least one false alarm is 1 − 0.358 = 64%. That is why a lone “winner” among many tries deserves a fresh test before anyone celebrates.
How the interactives work
The machine. Commute times are simulated with mean 30 minutes and standard deviation σ, from a normal distribution, a skewed one (an exponential, shifted so its mean is 30, with skewness 2), or an even mix of two normals at 30 ± 0.8σ with spread 0.6σ. Each sample gives Ȳ ± (critical value) × (s or σ)/√n, with critical values from the t or normal distribution in the course’s shared statistics library.
The dartboard. The population is two independent standard normals centered on the bullseye. The guess for shrinking is (0.55, 0.45). A volunteer is accepted with probability 1/(1 + e−2x), where x is the first number. The test is a z test with known σ: under the true effect d, the statistic is normal with mean d√n and standard deviation 1, so power, p-values and β are exact normal areas.
How it was checked (node tests, 40,000 intervals per case):
- Normal population, n = 5: t intervals cover 95.05%; “z with s” 87.6% (exact: 87.84%); n = 3, 81.1% (exact 81.09%). At 99% with n = 30: 99.09%. At 50%: 50.2%.
- Skewed population with the t method: 88.5% at n = 5, 92.0% at 20, 93.5% at 50, 94.0% at 100, 94.6% at 200.
- Darts, 20,000 rounds: the sample mean’s spread matches √(2/n) (0.283 at n = 25); shrinking halfway gives bias 0.355 (formula 0.355); the volunteers’ bias is 0.606 (exact by numerical integration: 0.6057).
- Power for d = 0.3 and n = 88: 80.35%, and 100,000 simulated studies rejected 80.37% of the time. With no effect, 4.98% rejected (α = 5%).
Check yourself
An estimator is unbiased if…
Show the answer
Unbiased means E(θ̂) = θ: right on average, not every time. Smallest variance is efficiency.
To cut a confidence interval’s width in half, you need about…
Show the answer
The width is proportional to 1/√n, so halving it takes four times as much data.
A p-value of 0.01 means…
Show the answer
The p-value is computed assuming H₀. It is not the probability of H₀ and says nothing about the size of the effect.
Rejecting a true null hypothesis is…
Show the answer
A false alarm is a type I error; you set its probability with α. Missing a real effect is a type II error.
Which change raises the power of a test?
Show the answer
More data shrinks the standard error, so a real effect stands out more. A smaller α makes rejection harder, which lowers power.
With 2 million observations, an A/B test finds that a new button raises sales by 0.01% with p = 0.001. This is…
Show the answer
Huge samples make tiny effects significant. Whether 0.01% is worth the change is a practical question.
Sources
- OpenStax, Introductory Statistics 2e: Ch. 8 Confidence Intervals and Ch. 9 Hypothesis Testing with One Sample.
- Penn State, STAT 415 Introduction to Mathematical Statistics (estimators, confidence intervals, tests and power).
- MIT OpenCourseWare, 18.05 Introduction to Probability and Statistics (Spring 2022).
- J. M. Wooldridge, Introductory Econometrics: A Modern Approach, 7th ed. (Cengage, 2020), Math Refresher C, Fundamentals of Mathematical Statistics (Appendix C in earlier editions). J. H. Stock and M. W. Watson, Introduction to Econometrics, 4th ed., Chapter 3.
- R. L. Wasserstein and N. A. Lazar, “The ASA Statement on p-Values: Context, Process, and Purpose,” The American Statistician 70 (2016).
- R. A. Fisher, Statistical Methods for Research Workers (1925). J. Neyman and E. S. Pearson, “On the Problem of the Most Efficient Tests of Statistical Hypotheses,” Philosophical Transactions of the Royal Society A 231 (1933). J. Neyman, “Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability,” Philosophical Transactions of the Royal Society A 236 (1937).