Econometrics · Chapter 5

Inference in Regression

A regression slope is a single number from a single sample. How much would it change with a different sample? Here you get to find out the way statisticians check their own formulas: by Monte Carlo. Set up a world where you know the true slope, draw thousands of samples, and watch the estimates pile up. Then break the assumptions one at a time and see which mistakes move the pile off the truth and which only make the standard errors lie.

The world

The truth bends (not linear) Noise grows with hours (heteroskedastic) Ability matters and is left out Classmates share luck (correlated errors)

Each sample

Students per samplen = 100

Noiseσ = 8 points

Hours studied range0.0 to 10.0 hours

Standard errors

Live numbers

Missions

What you’re seeing

The world. In this simulated school, the true relationship is score = 50 + 4 × hours + u: every hour of study adds 4 points, and u is everything else, with standard deviation σ. Only the simulation knows that the slope is 4. Each sample is a fresh class of n students.

The left panel shows the latest sample and its OLS line (pink), next to the true line (gold). Slow it down with “One at a time” and watch the pink line wobble from sample to sample.

The histogram collects the slope β̂₁ from every sample: the sampling distribution of the estimator, which in real life you never get to see. If OLS is unbiased, the pile centers on the gold line. The pink dashed curve is what the standard error formula claims the spread is; when it matches the blue pile, the standard errors are telling the truth.

The gauge counts how often the 95% confidence interval from each sample, β̂₁ ± t* × se, covers the true 4. Honest intervals cover about 95% of the time.

The assumptions

Wooldridge numbers the assumptions for simple regression SLR.1 to SLR.5. Each one buys something, and the switches in the lab break them one at a time.

Assumption	In words	What it buys	Broken in the lab by
SLR.1 Linear in parameters	`y` = β₀ + β₁`x` + `u` in the population	Unbiasedness: E(β̂₁) = β₁	“The truth bends”
SLR.2 Random sampling	Each observation is an independent draw		“Classmates share luck” (this hurts the standard errors, not the bias)
SLR.3 Variation in `x`	Not every `x`_i is the same		Shrink the hours range: the pile spreads out
SLR.4 Zero conditional mean	E(`u` \| `x`) = 0: nothing in `u` is related to `x`		“Ability is left out”
SLR.5 Homoskedasticity	Var(`u` \| `x`) = σ², the same for every `x`	The usual variance formula, and the Gauss–Markov theorem	“Noise grows with hours”
Normality (MLR.6 in Wooldridge)	`u` is normal and independent of `x`	Exact t distributions in small samples	“Skewed” and “Fat tails”

Why OLS is unbiased

Substitute the model y_i = β₀ + β₁x_i + u_i into the formula from Chapter 4. Because Σ(x_i − x̄) = 0 and Σ(x_i − x̄)x_i = SST_x, everything but the errors cancels:

β̂₁ = β₁ + Σ(x_i − x̄) u_i ÷ SST_xthe estimate = the truth + a weighted average of the errors, with SST_x = Σ(x_i − x̄)²

Take expectations given the x’s. Under SLR.4 each E(u_i | x) = 0, so the second term averages to zero and E(β̂₁) = β₁. Notice what was not needed: constant variance, independence between observations, or normality. That is why heteroskedasticity and correlated errors leave the pile centered on 4. But if something in u moves with x, like ability that is higher among students who study more, the second term doesn’t average out, and the pile shifts. That is omitted-variable bias, the subject of Chapter 6.

How much does β̂₁ wobble?

Add homoskedasticity (SLR.5) and independent observations, and the variance of the weighted average of errors is easy to compute:

Var(β̂₁) = σ² ÷ SST_x se(β̂₁) = σ̂ ÷ √SST_x, σ̂² = SSR ÷ (n − 2)more noise (σ) means more wobble; more spread in x, or more observations, means less

SST_x is about n × Var(x), so the standard error shrinks like 1/√n, just like the standard error of a mean in Chapter 3. In the lab’s default world, with σ = 8 and hours spread evenly from 0 to 10 (variance 100/12), a sample of 100 gives a standard deviation of 8/√(100 × 100/12) = 0.28; a sample of 400 halves it to 0.14.

We don’t know σ, so we estimate it from the residuals. We divide SSR by n − 2, not n, because the residuals had to satisfy two equations (they sum to zero and are uncorrelated with x), which uses up two degrees of freedom. With that correction σ̂² is unbiased for σ².

Worked example: the five students from Chapter 4

The fitted line was ŷ = 46.2 + 5.6 × hours, with SSR = 34.4 and SST_x = 10. Then σ̂² = 34.4/(5 − 2) = 11.47, σ̂ = 3.39, and se(β̂₁) = 3.39/√10 = 1.07.

Test H₀: β₁ = 0. t = 5.6/1.07 = 5.23 with 3 degrees of freedom; the two-sided p-value is 0.014. The 95% interval uses t* = 3.182: 5.6 ± 3.182 × 1.07 = 2.2 to 9.0 points per hour. Significant, but with five students the interval is wide.

t-tests and confidence intervals for a slope

Under SLR.1 to SLR.5 plus normal errors, the standardized slope has an exact t distribution with n − 2 degrees of freedom:

(β̂₁ − β₁) ÷ se(β̂₁) ~ t_n−2 95% CI: β̂₁ ± t* × se(β̂₁)to test H₀: β₁ = b, compute t = (β̂₁ − b)/se(β̂₁) and compare it with the critical value; the most common test is b = 0, “does x matter at all?”

Slope test calculator

Estimated slope β̂₁ Standard error Observations n Null value of β₁

Worked example: do ads pay for themselves?

Thirty stores’ weekly sales (in $1,000s) are regressed on their weekly ad spending (in $1,000s): β̂₁ = 2.35 with se = 0.82 (the calculator’s starting numbers). Each $1,000 of ads goes with $2,350 more sales.

Does advertising matter at all? t = 2.35/0.82 = 2.87 with 28 degrees of freedom, p = 0.008. Reject β₁ = 0.
Does a dollar of ads bring back more than a dollar of sales? Test β₁ = 1: t = (2.35 − 1)/0.82 = 1.65, p = 0.11. Not significant at 5%: the data can’t rule out break-even.
The 95% interval is 2.35 ± 2.048 × 0.82 = 0.67 to 4.03. It contains 1, matching the second test.

And remember Chapter 1: stores that advertise more may differ in other ways, so this slope is causal only if SLR.4 holds. Nothing here is financial advice.

The Gauss–Markov theorem: OLS is BLUE

OLS is a linear estimator: β̂₁ = Σ w_iy_i with weights w_i = (x_i − x̄)/SST_x that depend only on the x’s. Many other linear estimators are unbiased too. The theorem says OLS wins:

Under SLR.1 to SLR.5, OLS is the Best Linear Unbiased Estimatorno other estimator that is linear in y and unbiased has a smaller variance (Wooldridge, Theorem 3.4, for multiple regression)

The race

Noise grows with hours

Goals

The contestants. “Two halves” (an idea of Abraham Wald’s) sorts the students by hours, and divides the difference in average scores between the top and bottom halves by the difference in average hours. “Two endpoints” draws the line through the students with the fewest and the most hours. Both are linear in the scores and unbiased, and both are beaten by OLS when the noise is constant.

When OLS loses. Let the noise grow with hours. OLS stays unbiased but is no longer best: weighted least squares, which gives quiet observations more weight (weights 1/Var(u | x)), has a much smaller spread. In practice the weights are rarely known, so the usual fix is to keep OLS and use robust standard errors, which are valid whatever the pattern of the variance (Chapter 8).

Try this: In the main lab, press “Growing noise”. The pile still centers on 4, but it is wider than the pink curve, and the gauge drops below 90%. Switch the standard errors to “Robust”: the pink curve widens to match, and coverage comes back near 95%. The estimates didn’t change at all; only the honesty of the standard errors did.

Try this: Press “Classrooms”. Students in a classroom share part of their study habits and part of their luck, so 100 students carry much less independent information than 100 strangers. Classic standard errors pretend otherwise and cover only about 61% of the time. Clustered standard errors fix it, but they need many classrooms: with 10 they reach about 90%, with 50 about 94%.

When normality matters, and when it doesn’t

Not for unbiasedness or BLUE. Neither result uses normality. The normality assumption only buys the exact t distribution of the t statistic, which matters in small samples.

Not much in large samples. β̂₁ is a weighted average of many independent errors, so the central limit theorem makes it approximately normal whatever the errors look like. Then the t statistic is approximately standard normal, and the usual tests and intervals work. Wooldridge calls this asymptotic normality (Chapter 5 of his book). In the lab, skewed or fat-tailed errors with 100 students still give about 95% coverage; with spread-out x values, even 10 students do well. Small samples with a few extreme x values or very heavy tails are where it can fail.

Common mistake: “Regression requires normally distributed data”

Neither x nor y needs to be normal, and even the errors needn’t be for OLS to be unbiased and best linear unbiased. What really matters is SLR.4, that the errors are unrelated to x, which no amount of data or normality can fix. A related mistake is to treat “significant” as “causal”: the left-out-ability world produces a very significant, very precise, wrong slope.

A short history, and the people

Gauss and Markov. Carl Friedrich Gauss proved in the 1820s that least squares gives the smallest variance among linear unbiased estimators. The Russian mathematician Andrey Markov presented the result again in the early 1900s, and the theorem now carries both names.

Small samples. Gosset’s t distribution (1908) and R. A. Fisher’s work in the 1920s gave exact tests for regression coefficients with normal errors, the tests every regression program still prints.

Honest standard errors. In 1980 the econometrician Halbert White published standard errors that stay valid under heteroskedasticity of unknown form, now used routinely. Brent Moulton showed in 1990 how badly classic standard errors can mislead when observations share group-level shocks, and clustered standard errors became standard practice in applied economics.

Monte Carlo. The method of learning about a formula by simulating thousands of random samples was developed at Los Alamos in the 1940s by Stanislaw Ulam and John von Neumann, and Nicholas Metropolis named it after the casino in Monaco. Econometricians use it to check every new estimator.

Where you see it

Business. A retailer testing a promotion across stores has customers clustered within stores and stores within regions; standard errors that ignore the clustering make weak results look strong. Analysts routinely report robust or clustered standard errors for that reason.

Policy. When a policy changes for a whole state, every person in the state shares the change. In 2004 Marianne Bertrand, Esther Duflo and Sendhil Mullainathan showed that many published studies of such policies had standard errors that were far too small because they ignored correlation over time within states.

Science. Laboratory and field experiments use the same t-tests on slopes, and the same Monte Carlo checks of new methods before they are trusted.

Reading the news. When a study reports that something is “linked” to something else with p < 0.05, the questions from this chapter apply: is the slope unbiased (what’s in u?), and are the standard errors honest (are the observations independent)?

Practice

Standard error from scratch. A regression with n = 32 has SSR = 450 and SST_x = 200. Find σ̂ and se(β̂₁).
Show the answer
σ̂² = 450/(32 − 2) = 15, σ̂ = 3.87. se(β̂₁) = 3.87/√200 = 0.274.
A t-test. With n = 27, β̂₁ = 0.80 and se = 0.25. Test H₀: β₁ = 0 at 5%, two-sided, and give a 95% interval.
Show the answer
t = 0.80/0.25 = 3.20 with 25 degrees of freedom, p = 0.004: reject. t* = 2.060, so the interval is 0.80 ± 0.52, or 0.28 to 1.32.
Which assumption? For each, say whether OLS stays unbiased and whether classic standard errors stay valid: (a) the error variance is larger for high-income households; (b) people who choose more of x also have more of something in u; (c) the errors are skewed and n = 2,000; (d) the data are 40 students from each of 25 classrooms.
Show the answer
(a) Unbiased; classic standard errors invalid (use robust). (b) Biased: SLR.4 fails. (c) Unbiased, and with 2,000 observations the usual inference is fine. (d) Unbiased if SLR.4 holds, but classic standard errors are too small; cluster by classroom.
Designing for precision. With σ = 10 and x spread evenly between 0 and 20 (variance 400/12), how many observations give se(β̂₁) ≈ 0.1? What if you could choose half your x’s at 0 and half at 20?
Show the answer
se ≈ σ/√(n Var(x)) = 10/√(n × 33.3) = 0.1 needs n × 33.3 = 10,000, so n ≈ 300. With half at each end, Var(x) = 100, so n = 100 is enough. Spreading x out buys precision, though with only two values you can no longer see whether the relationship is a straight line.
Unbiased, step by step. Show that Σ(x_i − x̄)y_i = β₁ SST_x + Σ(x_i − x̄)u_i, and conclude that β̂₁ = β₁ + Σ(x_i − x̄)u_i/SST_x.
Show the answer
Plug in y_i = β₀ + β₁x_i + u_i: Σ(x_i − x̄)β₀ = 0, Σ(x_i − x̄)β₁x_i = β₁Σ(x_i − x̄)² = β₁SST_x, and the error term stays. Since β̂₁ = Σ(x_i − x̄)y_i/SST_x (the ȳ term drops out for the same reason β₀ does), divide by SST_x.
The variance formula. Using the result of problem 5, independent observations and Var(u_i | x) = σ², show Var(β̂₁) = σ²/SST_x. Which step fails under heteroskedasticity, and what replaces it?
Show the answer
Var(Σw_iu_i) with w_i = (x_i − x̄)/SST_x and independent errors is Σw_i² Var(u_i) = σ² Σ(x_i − x̄)²/SST_x² = σ²/SST_x. Under heteroskedasticity Var(u_i) = σ_i² can’t be pulled out, leaving Σ(x_i − x̄)²σ_i²/SST_x². White’s robust standard error estimates it by putting the squared residuals û_i² in place of σ_i².
Another unbiased estimator. Show that the “two endpoints” slope (y_b − y_a)/(x_b − x_a) is linear in the y’s and unbiased under SLR.1 and SLR.4. Why does Gauss–Markov say it can’t beat OLS?
Show the answer
It is a weighted sum of y’s with weights ±1/(x_b − x_a) on two observations and 0 on the rest: linear. Its expected value given the x’s is (β₁x_b − β₁x_a)/(x_b − x_a) = β₁: unbiased. Under SLR.1–SLR.5, OLS has the smallest variance in that class, so this estimator’s variance, 2σ²/(x_b − x_a)², can be no smaller than σ²/SST_x. It throws away the other n − 2 observations.

How the interactives work

The world. Hours are spread evenly over the chosen range; score = 50 + 4 × hours + σe, where e has mean 0 and variance 1 and is normal, a centered exponential (skewed) or a t with 3 degrees of freedom scaled to variance 1 (fat tails). The switches: “bends” adds −0.1 × hours²; “growing noise” multiplies e by (0.1 + t)², where t runs from 0 to 1 across the hours range, rescaled so the average variance is still σ²; “ability” adds 2 × (0.5 × (hours − 5) + v) with v standard normal; “classrooms” groups students in tens (fives when n < 50), with 70% of the hours range and half the error variance shared within a classroom. All data are simulated with a known truth.

The estimates. Each sample is fitted with the closed-form OLS formulas. Classic standard errors use σ̂²/SST_x; robust ones are White’s HC1, Σ(x_i − x̄)²û_i²/SST_x² × n/(n − 2); clustered ones sum (x_i − x̄)û_i within each classroom, with the usual small-sample factor and G − 1 degrees of freedom. The calculator uses the t distribution with n − 2 degrees of freedom.

How it was checked (node tests):

The closed forms match the course library’s matrix OLS in 300 random samples: coefficients to 8 × 10⁻¹², classic, robust and clustered standard errors to 10⁻¹² (relative), with the same degrees of freedom.
20,000 samples of 100 per world. All assumptions: mean 4.000, sd 0.281, average se 0.279 (theory 0.277), coverage 94.9%. Bending truth: mean 3.000 (the average slope, as theory predicts), coverage of 4 just 6%. Growing noise: mean 3.997, sd 0.339, classic se 0.276, coverage 88.9%; robust se 0.333, coverage 94.1%. Ability left out: mean 4.996 (theory 5.0); at n = 500, 5.000 with coverage 0%. Classrooms: sd 0.849, classic se 0.373, coverage 61.1%; clustered, 90.4% with 10 classrooms and 94.1% with 50. Skewed errors at n = 100: 95.0%.
The race, 20,000 samples of 50: with constant noise, sd 0.398 (OLS), 0.465 (two halves), 1.18 (two endpoints), all centered on 4; with growing noise, OLS 0.483 and weighted least squares 0.163.
Five-student example: se 1.0708, p = 0.0136, the same as the library’s regression output.

Check yourself

Which assumption is needed for OLS to be unbiased?
Show the answer
SLR.4 (with SLR.1 to SLR.3). Homoskedasticity and normality matter for the variance and for exact inference, not for unbiasedness.
The noise in y grows with x. What happens to OLS?
Show the answer
Heteroskedasticity leaves the estimates centered on the truth but breaks the usual variance formula; use robust standard errors.
Doubling the spread of x (its standard deviation), with the same n and σ, makes se(β̂₁)…
Show the answer
SST_x grows fourfold, and se = σ/√SST_x halves.
“BLUE” in the Gauss–Markov theorem means OLS has the smallest variance among…
Show the answer
Best Linear Unbiased Estimator. A biased or nonlinear estimator could have smaller variance, and under heteroskedasticity weighted least squares beats OLS.
With 5,000 observations and clearly skewed errors, the usual t-test for a slope is…
Show the answer
In large samples β̂₁ is approximately normal whatever the error distribution, so t-tests and intervals work.
A study of 2,000 students from 20 schools uses classic standard errors. The likely problem is…
Show the answer
Correlated errors within schools make the effective sample smaller than 2,000. Cluster by school (and 20 clusters is on the low side).

Sources

J. M. Wooldridge, Introductory Econometrics: A Modern Approach, 7th ed. (Cengage, 2020): Chapter 2, section 2.5 (unbiasedness and variances of OLS); Chapter 3, section 3.5 (the Gauss–Markov theorem); Chapter 4 (MLR.6 and t tests); Chapter 5 (asymptotic normality); Chapter 8 (heteroskedasticity-robust inference).
J. H. Stock and M. W. Watson, Introduction to Econometrics, 4th ed. (Pearson, 2019), Chapter 5 and its appendix on the Gauss–Markov theorem.
Penn State, STAT 501 Regression Methods: Lesson 2, SLR Model Evaluation and Lesson 4, SLR Model Assumptions.
MIT OpenCourseWare, 14.32 Econometrics (Spring 2007) and 14.382 Econometrics (Spring 2017).
H. White, “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity,” Econometrica 48 (1980). B. R. Moulton, “An Illustration of a Pitfall in Estimating the Effects of Aggregate Variables on Micro Units,” Review of Economics and Statistics 72 (1990). M. Bertrand, E. Duflo and S. Mullainathan, “How Much Should We Trust Differences-in-Differences Estimates?” Quarterly Journal of Economics 119 (2004).
A. C. Cameron and D. L. Miller, “A Practitioner’s Guide to Cluster-Robust Inference,” Journal of Human Resources 50 (2015).

I finished this chapterSaved on this device only.