Econometrics · Chapter 9
Specification, Measurement Error and Outliers
Real data are messy. Incomes are misreported, one wild data point can swing a whole line, and a flexible enough model can fit last year’s numbers perfectly and next year’s terribly. Here you’ll blur a variable with noise and watch the slope shrink toward zero by an amount you can predict, drag an outlier around until it bends the regression to its will, and turn up a polynomial’s degree until it memorizes noise. Along the way: how to spot a wrong functional form, and how to choose among models.
Measurement error
Live numbers
Missions
What you’re seeing
Permanent and measured income. Milton Friedman argued that people base their spending on their long-run, “permanent” income, not on this year’s ups and downs. In the simulation each household has a true permanent income (gold dots) and spends 8 + 0.90 × that, in thousands of dollars, plus a little noise. But a survey records this year’s income, which adds a random “transitory” part. Each blue dot is what the survey sees; the thin lines show how far the measurement error moved it.
The slope shrinks. Add noise to income and the cloud stretches sideways but not up, so the fitted green line flattens. It flattens by a predictable factor: the reliability ratio λ, the share of measured income’s variance that is real. The pink dotted line is λ × 0.90, and the green fit hugs it. Add noise to consumption instead and the cloud only stretches up and down: the slope stays right, just noisier.
The slope meter. The dot and white bar are this sample’s slope and 95% interval. Run 1,000 samples and the green histogram shows where slopes land: around λβ₁, not around the truth. Notice that the interval from a single sample is confidently wrong when λ is small.
Measurement error in x and in y
Suppose the true model is y = β₀ + β₁x* + u, but we observe x = x* + e. In the classical errors-in-variables setup, the measurement error e has mean zero and is uncorrelated with the true x* and with u. Substitute x* = x − e:
The new error contains −β₁e, and e is part of x, so the error is correlated with the regressor. OLS is biased and inconsistent:
Noise in y is different. If y = y* + v with v uncorrelated with x, the error just becomes u + v: no bias, only bigger standard errors.
With several regressors, error in one of them attenuates its own coefficient and can bias the others in either direction. And if the measurement error is not classical (people who earn more under-report more, say), the bias can go either way.
Suppose the variance of permanent income across households is 225 (a standard deviation of $15,000) and the transitory part adds variance 75. Then λ = 225/(225 + 75) = 0.75. If households truly spend $0.90 of each extra permanent dollar, a regression on measured income finds about 0.75 × 0.90 = 0.675. That is the pattern Friedman explained in 1957: in cross sections of households, consumption rises with measured income by less than the long-run spending share implies, partly because a lot of the measured income differences are temporary.
If a validation study shows that self-reported schooling has a reliability ratio of 0.90, and a regression of log wages on self-reported schooling (alone) finds 0.072, then the attenuation-corrected estimate is 0.072/0.90 = 0.080. With other regressors that are correlated with schooling the correction is larger, because the other regressors soak up part of schooling’s real variation and leave relatively more of the noise. A second, independent report of the same variable can also serve as an instrument (Chapter 10).
Proxy variables
Chapter 6’s omitted ability can sometimes be partly fixed with a proxy: an observed variable related to the unobserved one, such as an IQ score for ability. Put the proxy in the regression. For this to remove the bias, once the proxy is in, the rest of the unobserved variable must be unrelated to the other regressors; and the proxy must not belong in the true equation itself. A proxy that captures part of ability usually shrinks the bias without eliminating it.
Proxies differ from measurement error. Measurement error is about the variable you want, recorded badly. A proxy is a different variable standing in for one you can’t see. Putting a noisy measure of ability in a wage regression is both: a proxy, measured with error.
A wrong functional form, and the RESET test
If the truth curves and you fit a line, the model is misspecified, and the slope you report averages over a relationship that changes. Residual plots are the first check (Chapter 7). James Ramsey’s RESET test (1969) is a formal one: add powers of the fitted values, ŷ² and ŷ³, to the regression and F-test them. If they help, the original model missed some nonlinearity.
RESET tells you that something is wrong, not what. It has no power against omitted variables that enter linearly, and it is not a test for heteroskedasticity. In the polynomial lab below, the RESET p-value for a straight line is tiny; for degree 3 or more it usually isn’t.
Outliers, leverage and influence
An outlier is a point far from the rest. Whether it matters depends on where it is. A point with an unusual x has high leverage: the line has to pass near it. A point with a big residual but an ordinary x pulls the line up or down a little but can’t tilt it much. Influence combines the two. Drag the points below.
Outlier lab
For simple regression with n points and k = 2 parameters:
Cook’s distance measures how much all the fitted values move when point i is dropped. Common flags are D > 4/n (look closer) and D > 1 (strongly influential).
In the lab’s starting data, 16 points give a slope of about 0.84. Press “Add a high-leverage outlier”: a point at x = 14, far right of the others, with y = 3. Its leverage is 0.42 (the average is 2/17 = 0.12), its studentized residual −3.6, and its Cook’s distance about 4.6. The slope falls to about 0.40. Least absolute deviations, which minimizes Σ|ûi| instead of Σûi², gives about 0.80: it shrugs the outlier off.
What to do. First check whether the point is a data error (a misplaced decimal, a wrong unit). If it is real, report results with and without it, and consider a robust method such as least absolute deviations, which estimates the conditional median rather than the mean. Taking logs often tames extreme values too.
Dropping inconvenient data until the result looks nice is a way to fool yourself. Some of the most important observations are the extreme ones: the biggest firm, the deepest recession. Fix true errors, but keep real observations, and be open about how much the conclusions depend on a few points. A second mistake runs the other way: assuming a big residual is dangerous. A point in the middle of the x range with a huge residual has low leverage and small influence. Try mission “Loud but harmless.”
Multicollinearity and the variance inflation factor
Chapter 6 showed Var(β̂j) = σ² / [SSTj(1 − Rj²)]. The factor 1/(1 − Rj²) is the variance inflation factor: how much larger the variance is than it would be if xj were uncorrelated with the other regressors. Software reports it for each variable. A common rule of thumb calls VIF above 10 a concern, but there is no magic cutoff: a VIF of 20 with a huge sample can still give a precise estimate, and high VIFs among control variables don’t hurt the coefficient you care about.
In a house-price regression, regressing square footage on the number of rooms and bathrooms gives R² = 0.95. VIF = 1/0.05 = 20, so the standard error on square footage is √20 ≈ 4.5 times what it would be with uncorrelated regressors. If the coefficient is still significant, fine. If not, the data simply can’t separate size from rooms; more data, or a different question (“the effect of size, letting rooms change too”), helps more than deleting variables that belong.
Choosing among models: fit, penalties and out-of-sample tests
A more flexible model always fits the data it was estimated on at least as well. The question is whether it predicts new data better. Fit a polynomial to 20 training points from a smooth curve and judge it on 300 fresh points.
Polynomial lab
Penalized fit. With SSR from a model with k parameters fitted to n points (lower is better):
Hirotugu Akaike’s criterion (1974) aims at good prediction; Gideon Schwarz’s (1978) charges more per parameter once n > 7 (ln n > 2) and picks the true model with probability approaching 1 as n grows, if it is among the candidates. Adjusted R² charges the least.
Out-of-sample checks. The most direct test is to hold back data the model never saw, or to rotate which part is held back: k-fold cross-validation. In 200 simulated training sets of 20 points, test error was lowest at degree 3 to 5 in 177 of them. BIC chose degree 3 to 5 in 142, AIC in 73 and adjusted R² in only 55; with so few points, AIC and adjusted R² often chased the noise all the way to degree 12. Median test error rose from 0.29 at degree 4 to 2.5 at degree 12, while training error kept falling.
A short history
Least absolute deviations is older than least squares: the Croatian Jesuit scientist Ruđer Bošković (Roger Boscovich) proposed fitting a line by minimizing absolute deviations in the 1750s, studying the shape of the Earth.
Measurement error shaped one of economics’ big debates. Milton Friedman’s A Theory of the Consumption Function (1957) used the distinction between permanent and transitory income to explain why measured consumption responds less to measured income in cross sections than over the long run. It contributed to his 1976 Nobel memorial prize.
Diagnostics. James Ramsey published RESET in 1969. R. Dennis Cook introduced his distance in 1977. Hirotugu Akaike’s information criterion appeared in 1974 and Gideon Schwarz’s Bayesian criterion in 1978.
Where it shows up
Surveys. Self-reported income, hours, schooling and spending all carry measurement error, and recall gets worse with time. Attenuation is one reason estimated effects from surveys are often smaller than effects measured from administrative records.
Measuring the wrong thing. A measure can be precise and still be the wrong variable. Coca-Cola’s nearly 200,000 blind taste tests measured which sip people preferred, not how they felt about their brand (see the New Coke case). A proxy is only as good as its link to what you really care about.
Prediction and finance. A trading rule tuned on past data until it looks brilliant is the polynomial of degree 12: it fits the noise of the past. Tests on fresh data the rule never saw are the defense. Nothing on this page is financial or investment advice.
One big observation. In data on firms, cities or countries, one giant (the largest company, the capital city) can have enormous leverage. Report results with and without it.
Practice
Warm-up True income has variance 400 and measurement error has variance 100. If the true slope is 0.5, what does OLS on measured income estimate in large samples?
Show the answer
λ = 400/(400 + 100) = 0.8, so plim β̂₁ = 0.8 × 0.5 = 0.40.
Warm-up A firm’s sales figures are recorded with random typing errors (unrelated to anything else). The firm regresses sales on advertising. Is the advertising coefficient biased?
Show the answer
No: the error is in the dependent variable and unrelated to advertising, so it joins the regression error. The estimate is unbiased but less precise.
Core With n = 50, x̄ = 10 and SSTx = 800, what is the leverage of a point at x = 30? Is that high?
Show the answer
h = 1/50 + (30 − 10)²/800 = 0.02 + 0.5 = 0.52. The average leverage is k/n = 2/50 = 0.04, so this point is 13 times the average: very high.
Core A point has leverage 0.30 and studentized residual 2.5 in a simple regression. Compute Cook’s distance. Is it influential by the 4/n rule with n = 40? By D > 1?
Show the answer
D = 2.5² × 0.30 / [2 × 0.70] = 6.25 × 0.30 / 1.4 = 1.34. It exceeds 4/40 = 0.10 and also 1: strongly influential.
Core In a regression with three regressors, the R² from regressing x₁ on x₂ and x₃ is 0.90. What is the VIF for x₁, and by what factor is its standard error inflated?
Show the answer
VIF = 1/(1 − 0.90) = 10; the standard error is √10 ≈ 3.16 times as large as with uncorrelated regressors.
Core Two models fitted to n = 100 observations: model A has 3 parameters and SSR = 520; model B has 6 parameters and SSR = 480. Which does AIC prefer? BIC?
Show the answer
AIC: A = 100 ln(5.20) + 6 = 164.9 + 6 = 170.9; B = 100 ln(4.80) + 12 = 156.9 + 12 = 168.9. AIC prefers B. BIC: A = 164.9 + 3 ln 100 = 164.9 + 13.8 = 178.7; B = 156.9 + 6 × 4.61 = 156.9 + 27.6 = 184.5. BIC prefers A. The stronger penalty decides.
Challenge Show that classical measurement error in x attenuates the OLS slope. Start from y = β₀ + β₁x* + u and x = x* + e.
Show the answer
plim β̂₁ = Cov(x, y)/Var(x). Cov(x, y) = Cov(x* + e, β₀ + β₁x* + u) = β₁σ²x*, because e is uncorrelated with x* and u, and u is uncorrelated with x*. Var(x) = σ²x* + σ²e. So plim β̂₁ = β₁σ²x*/(σ²x* + σ²e) = λβ₁, with 0 < λ < 1 whenever σ²e > 0.
How the interactive works
Measurement error. True permanent income is normal with mean $60,000 and standard deviation $15,000; consumption = 8 + 0.90 × income + 4u (thousands). Measured income adds σₑ times a standard normal; measured consumption adds its own noise. Each sample keeps its random draws, so moving a noise slider stretches the same errors smoothly. The data are simulated with a known truth.
Outliers and polynomials. The outlier lab starts from 16 points around y = 2 + 0.8x with noise of standard deviation 0.8. Least absolute deviations is found exactly by checking every line through two data points, since an optimal line always passes through two of them. The polynomial lab draws 20 training points, one in each twentieth of [0, 4], and 300 test points from y = 2 + 1.2 sin(1.8x) − 0.25x plus noise; polynomials are fitted with Legendre polynomials (the same fits as powers of x, with better numerical accuracy).
How it was checked (node tests, all passed):
- Over 3,000 samples per case, the mean slope matched λβ₁: 0.4517 for λ = 0.5 (predicted 0.4500), 0.6228 for σₑ = 10 (0.6231), 0.2384 for σₑ = 25 (0.2382). With noise only in consumption: 0.9028 (truth 0.9000), but a spread of 0.123 against 0.019 with clean data.
- Cook’s distance from the leverage formula equaled the leave-one-out definition to 4 × 10⁻¹⁵; leverages summed to 2; 2,000 random nearby lines never beat the LAD line.
- RESET rejected a straight line for the curved data in 98.3% of 1,000 samples, and a correctly specified quadratic in 3.9% (nominal 5%). A VIF computed for two regressors with correlation 0.9 was 5.25 against the theoretical 5.26.
- The Legendre basis reproduced the raw-power fit to 10⁻⁹.
Check yourself
Classical measurement error in the regressor makes the OLS slope…
Show the answer
plim β̂₁ = λβ₁ with λ < 1: attenuation.
Classical measurement error in the dependent variable makes the OLS slope…
Show the answer
The error just adds to u; nothing correlates with x.
Which point is most likely to be influential?
Show the answer
Influence needs both leverage (unusual x) and a residual. A high-leverage point on the line doesn’t move it.
As you add polynomial terms, training error keeps falling. What happens to error on new data?
Show the answer
Past some point the extra flexibility fits noise, which doesn’t repeat in new data.
The RESET test adds what to the regression?
Show the answer
If ŷ² and ŷ³ are jointly significant, the model is missing some nonlinearity.
Sources
- J. M. Wooldridge, Introductory Econometrics: A Modern Approach, 7th ed. (Cengage, 2020), Chapter 9 (More on Specification and Data Issues): functional form and RESET, proxy variables, measurement error, outliers and least absolute deviations; Section 3-4 on the variance inflation factor; Section 6-3 on adjusted R² and choosing models.
- J. H. Stock and M. W. Watson, Introduction to Econometrics, 4th ed. (Pearson, 2019), Chapter 9 (Assessing Studies Based on Multiple Regression): errors-in-variables and other threats to validity.
- Penn State STAT 501, Lesson 11: Influential Points, Lesson 12: Multicollinearity and Lesson 10: Model Building.
- M. Friedman, A Theory of the Consumption Function (Princeton University Press for the National Bureau of Economic Research, 1957).
- J. B. Ramsey, “Tests for Specification Errors in Classical Linear Least-Squares Regression Analysis,” Journal of the Royal Statistical Society, Series B 31(2), 1969, 350–371.
- R. D. Cook, “Detection of Influential Observation in Linear Regression,” Technometrics 19(1), 1977, 15–18.
- H. Akaike, “A New Look at the Statistical Model Identification,” IEEE Transactions on Automatic Control 19(6), 1974, 716–723. G. Schwarz, “Estimating the Dimension of a Model,” Annals of Statistics 6(2), 1978, 461–464.