Econometrics · Chapter 6
Multiple Regression and Omitted Variables
Does a year of school raise your pay, or do people who stay in school longer earn more for other reasons? Here you’ll run that fight yourself. You control a world where wages depend on schooling and on an “ability” that the data never shows. Leave ability out and the regression gets the wrong answer, and you’ll see by exactly how much: the gap matches the omitted-variable formula to the last decimal. Then you’ll watch partialling out happen point by point, and turn a fitted plane in 3-D.
The hidden factor
Live numbers
Missions
What you’re seeing
The workers. Each dot is one simulated worker: years of schooling across, hourly wage up. Behind every dot is a number you never get to see in real data, the worker’s ability, shown here by color from blue (low) to gold (high). The true rule is wage = 8 + 2·school + β₂·ability + noise. So a year of school is truly worth exactly $2.00 an hour, the gold dotted line.
Two regressions. The pink line is the short regression: wage on schooling alone, which is all you could run if ability is missing from your data. The green line is the long regression: wage on schooling and ability together, drawn for a worker of average ability. The dashed lines are the long regression for workers one standard deviation below and above average. They’re parallel: within an ability level, school pays $2.
The formula box. The short slope always equals the long slope plus a correction, β̂₂ × δ̃₁: ability’s effect times how much ability rises with each year of school. The bars show that split on a dollar scale, and the table below shows which way the bias points.
The histograms. One sample can mislead by chance. Press “Run 1,000 samples” to redraw the world a thousand times. The green pile (long) centers on the truth. The pink pile (short) centers somewhere else, on $2 plus the bias. That gap is bias, not bad luck.
Holding other things constant
Simple regression has one explanatory variable. Real outcomes have many causes. Multiple regression puts several of them in one equation, so you can ask what happens when one changes and the others don’t:
Ordinary least squares (OLS) picks the estimates β̂₀, β̂₁, …, β̂k that make the sum of squared residuals, Σûi², as small as possible. The fitted equation ŷ = β̂₀ + β̂₁x₁ + … + β̂kxk gives predicted changes:
That is the meaning of each coefficient: the predicted change in y when x₁ rises by one unit and the other included variables stay fixed. Economists call this ceteris paribus, Latin for “other things being equal.”
Regression holds things fixed statistically, not physically. Nobody had to find two workers with identical experience; OLS compares everyone and sorts out the separate effects from how the variables move together in the data.
It can only hold fixed what is in the equation. Anything left out sits inside u. The key assumption for unbiased estimates is that the error has mean zero whatever the explanatory variables are:
If something in u moves with x₁, this fails, and β̂₁ picks up part of that something’s effect.
Suppose a regression on 1,000 workers (a made-up but realistic equation) gives
ŵage = −4.50 + 1.95·educ + 0.21·exper + 0.35·tenure
with wage in dollars per hour and the rest in years. Holding experience and tenure fixed, one more year of school predicts $1.95 more per hour. Now compare Ana (16 years of school, 5 years at her firm) with Ben (12 years of school, 2 years at his firm), both with 10 years of total experience. The predicted difference is 1.95 × 4 + 0.21 × 0 + 0.35 × 3 = 7.80 + 1.05 = $8.85 an hour. Over 2,000 hours a year, that is about $17,700.
Omitted-variable bias
Suppose the true model is y = β₀ + β₁x₁ + β₂x₂ + u, but x₂ isn’t in your data, so you run the short regression of y on x₁ alone and get slope β̃₁. (The tilde marks the short regression; hats mark the long one.) Let δ̃₁ be the slope from regressing the left-out x₂ on x₁. Then, in every sample:
Why it works. The short slope is Σ(xi1 − x̄₁)yi / Σ(xi1 − x̄₁)². Put the true model in for yi. The β₁ term comes back as β₁, the β₂xi2 term comes back as β₂ times the slope of x₂ on x₁ (that is δ̃₁), and the u term averages to zero under the zero-conditional-mean assumption. The identity with hats holds exactly in every sample; the simulation shows the two sides agree to about 10⁻¹³.
When there is no bias. Either the left-out variable doesn’t affect y (β₂ = 0), or it is uncorrelated with x₁ in the sample (δ̃₁ = 0). Leaving out a variable that matters is harmless if it doesn’t move with what you kept.
Which way it points. The sign of the bias is the sign of β₂ times the sign of the correlation between x₁ and x₂:
| corr(x₁, x₂) > 0 | corr(x₁, x₂) < 0 | |
|---|---|---|
| β₂ > 0 | positive bias (too high) | negative bias (too low) |
| β₂ < 0 | negative bias (too low) | positive bias (too high) |
In the classic case, ability raises pay (β₂ > 0) and more able people tend to stay in school longer (positive correlation), so the short regression overstates the return to schooling. Economists call this ability bias.
A short regression of hourly wage on schooling gives $2.60 per year. Suppose ability raises wages by $3.00 per standard deviation (β₂ = 3), and in this sample each extra year of school goes with 0.25 standard deviations more ability (δ̃₁ = 0.25).
- Bias = β₂ × δ̃₁ = 3.00 × 0.25 = $0.75.
- So the long regression would give about 2.60 − 0.75 = $1.85 per year of school.
- Over a 40-year career of 2,000 hours a year, the short regression overstates the value of one extra year by 0.75 × 2,000 × 40 = $60,000 (before any discounting).
Set ability’s effect to 3 and push the correlation until δ̃₁ in the readout is near 0.25; the formula box shows the same split.
Adding a variable that truly belongs removes bias. But adding one that doesn’t belong costs precision when it is correlated with your variable of interest: press “Irrelevant” and run 1,000 samples, and the green pile comes out wider than the pink one. Worse are bad controls: variables that are themselves outcomes of the thing you’re studying. To measure the pay effect of a college degree, don’t control for occupation, since the degree is part of how people get into higher-paying occupations. Holding occupation fixed would hide part of the effect you wanted to measure. Choose controls that are fixed before the cause you study, and that affect the outcome.
Partialling out: what “holding fixed” does inside OLS
How does OLS manage to hold experience fixed? The Frisch–Waugh–Lovell theorem gives a clear answer: β̂₁ in the multiple regression is the slope from a simple regression of y on the part of x₁ that the other regressors can’t explain. Step through it below with 60 simulated workers whose wage = 2 + 1.80·school + 0.50·experience + noise, where people with more schooling have had fewer years to gain experience.
Step through it
In symbols: regress x₁ on the other regressors and save the residuals r̂i1. Then
The residual r̂i1 is schooling with experience “partialled out”: the variation in school that has nothing to do with experience. Only that variation identifies β₁. This is why the simple slope (pink) and the multiple-regression slope (green) differ: the simple slope also picks up experience’s effect, because school and experience move together.
The theorem also explains what drives precision. The variance of β̂₁ is
where SST₁ = Σ(xi1 − x̄₁)² is the total variation in x₁ and R₁² is the R² from regressing x₁ on the other regressors. 1 − R₁² is the share of x₁’s variation left over after partialling out. Push the correlation slider to −0.9: the residuals in step 2 shrink to a thin band, and the slope in step 3 rests on very little variation.
Seeing it in 3-D: a plane, not a line
With two explanatory variables, the fitted equation is a flat plane floating over the floor of schooling and experience. Its tilt in the schooling direction is β̂₁; its tilt in the experience direction is β̂₂. Drag to turn it. Press “Look along experience” and the experience axis points straight at you: each slice of the plane at fixed experience is a line with slope β̂₁, which is what “holding experience fixed” looks like. Green sticks are positive residuals, red ones negative.
Turn the cube
These are the same 60 workers as in the partialling-out steps above; change the correlation there and the plane updates here.
R² and adjusted R²
R² is the share of the variation in y that the regression explains: R² = 1 − SSR/SST, where SSR = Σûi² and SST = Σ(yi − ȳ)². Adding a variable can never raise SSR, because OLS could always set its coefficient to zero. So R² never falls when you add a regressor, even a column of random numbers.
Adjusted R² charges for each extra regressor:
with k slope coefficients. It rises when a new variable’s |t| statistic is above 1 and falls when it is below 1.
With n = 50 and k = 3, a regression has R² = 0.400. Adjusted: 1 − 0.600 × 49/46 = 1 − 0.639 = 0.361.
Add two variables of pure noise and R² creeps up to 0.410. Adjusted: 1 − 0.590 × 49/44 = 1 − 0.657 = 0.343. The plain R² went up; the adjusted one went down.
A low R² doesn’t mean bias, and a high one doesn’t mean causation. In the simulation, ability bias appears with R² anywhere from 0.1 to 0.9. Whether β̂₁ is unbiased depends on what is in u, not on how big u is.
Multicollinearity
Perfect multicollinearity means one regressor is an exact linear combination of the others. Then OLS has no unique answer and software drops a variable or refuses. Examples: total ad spending together with TV spending and online spending, when total = TV + online; income in dollars and income in thousands of dollars; or a dummy variable for every category plus an intercept (Chapter 7’s “dummy-variable trap”). The fix is to drop one redundant variable, and to think about what the remaining coefficients now mean: with total and TV spending in the model, the TV coefficient measures moving a dollar from online to TV, holding the total fixed.
Near multicollinearity is high but not perfect correlation. It breaks nothing: OLS stays unbiased. It only makes the affected coefficients imprecise, through the 1/(1 − Rj²) factor in the variance, called the variance inflation factor (VIF).
If schooling and experience have R₁² = 0.90, the VIF is 1/(1 − 0.90) = 10, so the standard error of β̂₁ is √10 ≈ 3.2 times what it would be with uncorrelated regressors. At R₁² = 0.99 the VIF is 100 and the standard error is 10 times as large. You would need about 10 times (or 100 times) as many observations to get the precision back.
Chapter 9 returns to VIFs; Arthur Goldberger joked that the right name for near multicollinearity is “micronumerosity”: too little information in the data.
y = β₀ + β₁x₁ + … + βkxk + u. The x’s can be logs, squares or products (Chapter 7).
The data are a random sample of n observations from the population.
No regressor is constant or an exact linear combination of the others.
E(u | x₁, …, xk) = 0. Omitted variables that move with the x’s break it.
Var(u | x’s) = σ². With MLR.1–5, OLS is the best linear unbiased estimator (the Gauss–Markov theorem). Chapter 8 drops this one.
A short history
Least squares. Adrien-Marie Legendre published the method of least squares in 1805, in an appendix on fitting the orbits of comets. Carl Friedrich Gauss published his own account in 1809 and said he had used it since 1795, which started a long priority dispute. Both were fitting astronomical data with several unknowns at once: multiple regression before it had the name.
Regression and “other things equal.” Francis Galton gave us the word regression in the 1880s, from studying heights of parents and children. The statistician George Udny Yule then used least squares with several variables to study poverty in England, in an 1899 paper on whether more generous “outdoor relief” went with more pauperism once changes in population and age structure were held fixed. He was careful to say his results showed association, not proof of cause.
Partialling out. Ragnar Frisch and Frederick Waugh showed in 1933, in the first volume of Econometrica, that detrending each variable and then regressing gives the same coefficients as putting a time trend in the regression. Michael Lovell generalized the result in 1963. Frisch shared the first Nobel memorial prize in economics in 1969.
Ability bias. Whether schooling’s payoff is overstated because able people both study longer and earn more was a central question in labor economics after the 1950s. Zvi Griliches surveyed it in a 1977 Econometrica paper. Chapter 10 shows the instrumental-variables approach that later studies used to attack it.
Where it shows up
Pay gaps. A “raw” pay gap between two groups compares averages. An “adjusted” gap holds experience, hours and field fixed. The size of the adjusted gap depends on which controls go in, and some candidate controls, like job title, may be partly results of the very discrimination being measured: a bad-control problem.
Marketing. Stores advertise most in the weeks when sales are high anyway, such as before holidays. Regress sales on ad spending without holding the season fixed and the ad effect comes out too big: the season is the omitted variable, positively tied to both.
House prices. In a regression of price on square footage and the number of bedrooms, the bedroom coefficient can come out negative. That is not a mistake: holding square footage fixed, an extra bedroom means smaller rooms.
Firm comparisons. Costco pays well above most retailers and says its workers stay longer and serve customers better (see the Costco case). Comparing Costco with other chains can’t by itself prove that the pay causes the lower turnover: location, customers and who applies differ too. Each of those is a candidate omitted variable.
Health. If coffee drinkers also smoke more, a study that links coffee to heart disease without holding smoking fixed will blame coffee for some of smoking’s harm. Epidemiologists call omitted variables confounders.
Practice
Warm-up A fitted model is ŷ = 10 + 0.5x₁ − 2x₂. (a) What happens to ŷ if x₁ rises by 4 and x₂ stays fixed? (b) If x₁ rises by 4 and x₂ rises by 1?
Show the answer
(a) Δŷ = 0.5 × 4 = 2. (b) Δŷ = 0.5 × 4 − 2 × 1 = 0. The two changes cancel.
Warm-up A study regresses students’ test scores on class size, but family income is left out. Higher income raises scores, and richer districts tend to have smaller classes. Which way is the class-size coefficient biased?
Show the answer
Income’s effect on scores is positive (β₂ > 0), and income is negatively correlated with class size. Positive × negative = negative bias. The estimated class-size coefficient is pushed down, so big classes look more harmful than they really are.
Core The short regression of log wage on schooling gives β̃₁ = 0.12. In the long regression with an ability test score, β̂₂ = 0.50, and the regression of the test score on schooling gives δ̃₁ = 0.08. What is the long-regression coefficient on schooling?
Show the answer
β̃₁ = β̂₁ + β̂₂δ̃₁, so β̂₁ = 0.12 − 0.50 × 0.08 = 0.12 − 0.04 = 0.08. A third of the short-regression return was ability.
Core With n = 30, a regression with k = 4 slopes has R² = 0.52. Compute adjusted R². A fifth variable raises R² to 0.53. Does adjusted R² go up or down?
Show the answer
Before: 1 − 0.48 × 29/25 = 1 − 0.5568 = 0.443. After: 1 − 0.47 × 29/24 = 1 − 0.5679 = 0.432. It goes down, so the fifth variable’s |t| is below 1.
Core A marketing analyst regresses sales on total ad spending, TV spending and online spending, where total = TV + online. What goes wrong, and how should she fix it? After the fix, what does the TV coefficient mean?
Show the answer
Perfect multicollinearity: total is an exact linear combination of the other two, so OLS can’t separate the effects. Drop one of the three. If she keeps total and TV, the TV coefficient is the change in sales from moving one dollar from online to TV while keeping the total fixed. If she keeps TV and online, each coefficient is the effect of one more dollar on that channel with the other fixed.
Challenge In a regression of y on x₁ and x₂, σ̂ = 5, SST₁ = 400, and the R² from regressing x₁ on x₂ is 0.75. What is se(β̂₁)? What would it be if x₁ and x₂ were uncorrelated (same σ̂ and SST₁)?
Show the answer
se(β̂₁) = σ̂ / √[SST₁(1 − R₁²)] = 5 / √(400 × 0.25) = 5/10 = 0.50. With R₁² = 0: 5/√400 = 0.25. The correlation doubled the standard error (VIF = 4, √4 = 2).
Challenge Derive the omitted-variable formula. Start from β̃₁ = Σ(xi1 − x̄₁)yi / SST₁ and the true model yi = β₀ + β₁xi1 + β₂xi2 + ui.
Show the answer
Substitute: β̃₁ = [β₀Σ(xi1 − x̄₁) + β₁Σ(xi1 − x̄₁)xi1 + β₂Σ(xi1 − x̄₁)xi2 + Σ(xi1 − x̄₁)ui] / SST₁. The first sum is 0. The second is SST₁, giving β₁. The third over SST₁ is exactly δ̃₁, the slope of x₂ on x₁. So β̃₁ = β₁ + β₂δ̃₁ + Σ(xi1 − x̄₁)ui/SST₁. Taking expectations given the x’s, with E(u | x₁, x₂) = 0, the last term has mean zero: E(β̃₁) = β₁ + β₂δ̃₁.
How the interactive works
The world. Each worker gets an ability a from a standard normal distribution. Schooling is 13 + 2.5(r a + √(1 − r²) e) years, so its correlation with ability is exactly the slider value r. Wage = 8 + 2·school + β₂·a + 5u, with e and u standard normal. The population bias of the short slope is β₂ × cov(school, a)/var(school) = β₂r/2.5. The data are simulated with a known truth, so you can check every estimate against it. Real schooling comes in whole years and real wages can’t go below zero; the simulation skips both details to keep the formulas exact.
The estimates. All regressions use the site’s shared statistics library (ordinary least squares with classic standard errors). The Monte Carlo redraws the whole sample 1,000 times with new random numbers and records both slopes. The partialling-out and 3-D pieces share 60 workers with wage = 2 + 1.8·school + 0.5·experience + 4u, and experience negatively correlated with school (kept at zero or above).
How it was checked (node tests, all passed):
- The identity β̃₁ = β̂₁ + β̂₂δ̃₁ held in 300 random samples with a worst error of 9.5 × 10⁻¹⁴.
- Monte Carlo with 4,000 samples each: with n = 200, β₂ = 3, r = 0.6, the short slope averaged 2.7216 against the predicted 2.7200 (standard error 0.0025), and the long slope 2.0001. With n = 50, β₂ = 8, r = −0.7: short −0.2408 against −0.2400. With β₂ = 0 and r = 0.8 both were unbiased, but the long slope’s spread was 0.241 against 0.142 for the short one.
- Frisch–Waugh–Lovell: the step-3 slope equaled the multiple-regression coefficient in 200 samples to 9 × 10⁻¹³.
- Coefficients, standard errors, R² and adjusted R² matched NumPy’s least squares to 12 or more digits; se(β̂₁) matched σ̂/√[SST₁(1 − R₁²)] to 10⁻¹⁰.
Check yourself
In ŵage = −4.5 + 1.95·educ + 0.21·exper, what does 1.95 mean?
Show the answer
Each coefficient in a multiple regression is a ceteris paribus effect: the other included variables are held fixed.
The short regression of wage on school gives $2.80; the long one with ability gives $2.00. What must be true?
Show the answer
β̃₁ − β̂₁ = β̂₂δ̃₁ = 0.80 > 0, so β̂₂ and δ̃₁ have the same sign. The classic case is both positive. Sample size has nothing to do with this identity.
You leave out a variable that strongly affects y but is uncorrelated with x₁. The slope on x₁ is…
Show the answer
Bias = β₂δ₁, and δ₁ = 0. Leaving the variable out makes the error bigger (so standard errors are larger) but causes no bias.
You add a column of random numbers to a regression. What happens?
Show the answer
R² can never fall when a regressor is added. Adjusted R² rises only if the new variable’s |t| exceeds 1, which pure noise rarely does.
Which of these regressions suffers from perfect multicollinearity?
Show the answer
The two dummies add up to 1 for everyone, which is exactly the intercept column. Correlation of 0.95 is near, not perfect, collinearity; and age² is a nonlinear function of age, which is allowed.
Ability bias is $0.80 with samples of 200 workers. What happens with samples of 2,000?
Show the answer
Bias doesn’t depend on n. The spread shrinks like 1/√n, but the center stays at β₁ + β₂δ₁. The short-regression estimator is inconsistent.
Sources
- J. M. Wooldridge, Introductory Econometrics: A Modern Approach, 7th ed. (Cengage, 2020), Chapter 3 (Multiple Regression Analysis: Estimation): interpretation, “partialling out,” omitted-variable bias and its sign table, the variance formula and multicollinearity, the Gauss–Markov assumptions; Section 6-3 on adjusted R².
- J. H. Stock and M. W. Watson, Introduction to Econometrics, 4th ed. (Pearson, 2019), Chapter 6 (Linear Regression with Multiple Regressors): the omitted-variable bias formula and multicollinearity.
- J. D. Angrist and J.-S. Pischke, Mostly Harmless Econometrics (Princeton University Press, 2009), Section 3.2: regression and causality, omitted-variables bias and “bad control.”
- Penn State STAT 501, Lesson 5: Multiple Linear Regression and Lesson 12: Multicollinearity and Other Regression Pitfalls.
- MIT OpenCourseWare, 14.32 Econometrics (Spring 2007): course materials on multiple regression.
- R. Frisch and F. V. Waugh, “Partial Time Regressions as Compared with Individual Trends,” Econometrica 1(4), 1933, 387–401. M. C. Lovell, “Seasonal Adjustment of Economic Time Series and Multiple Regression Analysis,” Journal of the American Statistical Association 58, 1963, 993–1010.
- G. U. Yule, “An Investigation into the Causes of Changes in Pauperism in England, Chiefly During the Last Two Intercensal Decades,” Journal of the Royal Statistical Society 62(2), 1899, 249–295.
- Z. Griliches, “Estimating the Returns to Schooling: Some Econometric Problems,” Econometrica 45(1), 1977, 1–22.