Econometrics · Chapter 4
Simple Regression: Fitting a Line
Fitting a line through a cloud of points is the heart of econometrics. Here every mistake the line makes is drawn as a square, and ordinary least squares is the line that makes the total area of those squares as small as it can be. Drag the points and watch it refit. Draw your own line and try to beat it (you can’t, but you can tie). Then go back to 1886, when Francis Galton measured the heights of parents and their children and gave regression its strange name.
Data
Squares
Live numbers
Missions
What you’re seeing
The pink line is the ordinary least squares (OLS) line. For each point, the vertical gap between the point and the line is a residual, the part of y the line doesn’t explain. Each residual is drawn as a square, so a residual twice as big makes a square four times as big. OLS picks the one line with the smallest total area of squares.
The gold dashed line is yours. It starts flat at the average of y, which is the best you can do without using x at all. Drag its handles. The panel compares the two sums of squares; you can get as close as you like, but never below.
The gold cross marks the point of means, (x̄, ȳ). The OLS line always passes through it, however you move the points.
The bowl at the bottom plots the sum of squared residuals for every slope, using the best intercept for each. It is a parabola, and its lowest point is the OLS slope. Your line is the gold dot: it sits on the bowl when your line passes through the means, and above it otherwise.
Deriving the least squares line
From Chapter 1, the population model is y = β₀ + β₁x + u. We have a sample of n points (xi, yi). Any candidate line b₀ + b₁x leaves residuals yi − b₀ − b₁xi. Least squares chooses b₀ and b₁ to minimize the sum of their squares:
Step 1: set the derivatives to zero. Differentiating with respect to b₀ and b₁ and dividing by −2 gives the two first-order conditions:
In words: the residuals add up to zero, and they are uncorrelated with x. These are exactly the sample versions of the population assumptions E(u) = 0 and E(xu) = 0, which is how Wooldridge derives OLS as a “method of moments” estimator.
Step 2: solve. The first condition says ȳ = β̂₀ + β̂₁x̄, so β̂₀ = ȳ − β̂₁x̄: the line passes through the means. Put that into the second condition and rearrange (using Σ(xi − x̄) = 0):
Step 3: check it’s a minimum. SSR is a sum of squares, a bowl-shaped (convex) function of b₀ and b₁, so the point where the slopes are zero is the bottom. That is the bowl at the bottom of the interactive.
The fitted values are ŷi = β̂₀ + β̂₁xi and the residuals are ûi = yi − ŷi. Don’t confuse a residual û, which we compute, with the error u, which we never see: the residual is the gap to the estimated line, the error is the gap to the true one.
Hours studied x = 1, 2, 3, 4, 5 and exam scores y = 52, 60, 58, 70, 75. Then x̄ = 3 and ȳ = 63.
- Deviations: x − x̄ = −2, −1, 0, 1, 2 and y − ȳ = −11, −3, −5, 7, 12.
- Σ(x − x̄)(y − ȳ) = 22 + 3 + 0 + 7 + 24 = 56, and Σ(x − x̄)² = 4 + 1 + 0 + 1 + 4 = 10.
- β̂₁ = 56/10 = 5.6 points per hour; β̂₀ = 63 − 5.6 × 3 = 46.2.
- Fitted values 51.8, 57.4, 63.0, 68.6, 74.2; residuals 0.2, 2.6, −5.0, 1.4, 0.8. They add up to 0, as promised.
- SSR = 0.04 + 6.76 + 25 + 1.96 + 0.64 = 34.4.
How well does the line fit? R²
Split the variation in y into the part the line explains and the part it leaves over. With Wooldridge’s names (other books swap the letters, so check):
In the five-student example, SST = 121 + 9 + 25 + 49 + 144 = 348 and SSR = 34.4, so R² = 1 − 34.4/348 = 0.90. Your flat line at ȳ has SSR = SST, which is why it scores R² = 0: it uses no information from x.
A low R² doesn’t make a regression useless. Wages depend on so many things that education alone explains a small share of the differences between people, yet the slope can still be estimated precisely and matter a great deal.
A high R² doesn’t make it causal. The ice-cream town in Chapter 1 had a high R² and no causal effect at all. R² measures fit, not truth.
Units and interpretation
The slope is measured in units of y per unit of x: “5.6 points per hour of study.” The intercept is the predicted y when x = 0, which may or may not make sense (a person with zero years of schooling, a store with zero square feet). Change the units and the numbers change in a predictable way: measure sales in dollars instead of thousands of dollars and both β̂₀ and β̂₁ are multiplied by 1,000; measure x in minutes instead of hours and β̂₁ is divided by 60. R² doesn’t change at all.
The Bureau of Labor Statistics reports median weekly earnings in 2024 for full-time workers aged 25 and over by education. Giving each level a typical number of years of school:
| Education | Years (x) | Median weekly earnings (y) | Fitted ŷ |
|---|---|---|---|
| Less than high school | 10 | $738 | $650 |
| High school diploma | 12 | $930 | $934 |
| Some college, no degree | 13 | $1,020 | $1,076 |
| Associate’s degree | 14 | $1,099 | $1,219 |
| Bachelor’s degree | 16 | $1,543 | $1,503 |
| Master’s degree | 18 | $1,840 | $1,788 |
Here x̄ = 13.83 and ȳ = $1,195. The sample covariance is 1,161.8 and the sample variance of x is 8.167, so β̂₁ = 1,161.8/8.167 = $142 a week per extra year, and β̂₀ = 1,195 − 142.26 × 13.83 = −$773. R² = 0.97. The intercept, earnings with zero schooling, is far outside the data and means nothing on its own.
Three cautions. These are six group medians, not individual workers, so the high R² says nothing about how well schooling predicts any one person’s pay. The line misses a pattern: the jumps at degrees are bigger than at in-between years (Chapter 7 handles that with logs and dummy variables). And as Chapter 1 warned, this is an association; whether a year of school causes $142 more a week is a harder question.
Why it’s called “regression”
In the 1880s Francis Galton, a cousin of Charles Darwin, collected the heights of 205 sets of parents and their 928 grown children. To compare men and women, he multiplied every female height by 1.08. He found that unusually tall parents tended to have children who were tall, but less so; unusually short parents had children who were short, but less so. He called this “regression towards mediocrity,” by which he meant toward the average. The name stuck to the method.
“When Mid-Parents are taller than mediocrity, their Children tend to be shorter than they.”
Galton’s “mid-parent” is the average of the father’s height and the mother’s height times 1.08. He estimated that the children’s deviations from the average were about two-thirds of their mid-parents’.
Francis Galton, “Regression Towards Mediocrity in Hereditary Stature,” Journal of the Anthropological Institute of Great Britain and Ireland 15 (1886), pp. 246–263, as quoted by J. A. Hanley (2004).Families
Goals
What’s going on. A person’s height is partly inherited and partly luck: nutrition, health and the shuffle of genes. Parents who are very tall are, on average, tall partly by luck, and children don’t inherit their parents’ luck. So the children of tall parents are tall, but closer to the average. With a correlation of 0.5 between a parent’s height and a child’s, and equal spreads, the slope of the regression line is exactly 0.5: a parent 4 inches above average has children 2 inches above average, on average.
The paradox. If every generation regressed toward the middle, wouldn’t everyone end up the same height? No: look at the histograms. The children are just as spread out as the parents, because new luck spreads them out again. And regression runs backward too: tick “Pick children”, choose tall children, and their parents are closer to the average. The regression of parent on child (blue) is not the same line as child on parent (pink). In general the two slopes multiply to r².
Nothing pulls. Whenever two measurements are imperfectly correlated, the extremes of one will, on average, be less extreme in the other, purely because the luck that made them extreme doesn’t repeat. The danger is mistaking it for an effect. Send a consultant to your ten worst-performing stores this quarter and next quarter they will probably improve, consultant or not. Praise a salesperson after a record month and next month’s sales will probably dip. To know whether the consultant helped, you need a comparison group of bad stores that didn’t get one, the idea of Chapter 1.
A short history, and the people
Least squares. Adrien-Marie Legendre published the method of least squares in 1805, in an appendix to a book on the orbits of comets: choose the orbit that makes the sum of squared errors smallest. Carl Friedrich Gauss published it in 1809 and claimed he had been using it since 1795, setting off a famous priority dispute. Gauss also connected it to probability: least squares gives the most likely answer when errors follow the normal curve.
Regression and correlation. Galton’s studies of sweet peas and of human heights in the 1870s and 1880s turned least squares from a tool for astronomers into a way to describe relationships between variables. Karl Pearson formalized the correlation coefficient in the 1890s, and George Udny Yule applied regression to social and economic data around 1900, including studies of poverty relief in England.
Where you see it
Business. How much do sales rise with advertising, or fall with price? What does an extra square foot add to rent? A fitted line gives a first answer and a forecast, as long as you remember Chapter 1: the slope describes the data, and it is causal only under conditions we’ll study in Chapter 5 and beyond.
Sports. The rookie of the year who has a weaker second season, the star whose magazine cover is followed by a slump: much of that is regression to the mean. An extreme season usually includes extra luck that doesn’t come back.
Health and policy. People often try a remedy, or a program, when things are at their worst, and things then improve on their own. Studies without a comparison group can mistake regression to the mean for a cure.
Science. Edwin Hubble’s 1929 plot of galaxies’ speeds against their distances was summed up by a straight line, and its slope, now called the Hubble constant, measures how fast the universe is expanding.
Practice
By hand. Fit the OLS line to the points (1, 3), (2, 5), (3, 4), (4, 8).
Show the answer
x̄ = 2.5, ȳ = 5. Deviations: x: −1.5, −0.5, 0.5, 1.5; y: −2, 0, −1, 3. Σ(products) = 3 + 0 − 0.5 + 4.5 = 7; Σ(x − x̄)² = 2.25 + 0.25 + 0.25 + 2.25 = 5. β̂₁ = 7/5 = 1.4, β̂₀ = 5 − 1.4 × 2.5 = 1.5. So ŷ = 1.5 + 1.4x.
Residuals and R². For the line in problem 1, find the residuals, check that they sum to zero, and compute R².
Show the answer
Fitted: 2.9, 4.3, 5.7, 7.1. Residuals: 0.1, 0.7, −1.7, 0.9, which sum to 0. SSR = 0.01 + 0.49 + 2.89 + 0.81 = 4.2. SST = 4 + 0 + 1 + 9 = 14. R² = 1 − 4.2/14 = 0.70.
From summary numbers. For 50 coffee shops, the sample covariance of daily foot traffic and daily revenue is 2,400 (people × dollars), the variance of foot traffic is 1,600 (people²), average traffic is 300 people and average revenue $2,100. Find the regression line and interpret the slope.
Show the answer
β̂₁ = 2,400/1,600 = $1.50 per visitor; β̂₀ = 2,100 − 1.5 × 300 = $1,650. Each extra visitor goes with $1.50 more revenue, on average. (The intercept, revenue with zero visitors, is outside the data and shouldn’t be taken literally.)
Changing units. A regression of salary (in dollars) on experience (in years) gives β̂₀ = 42,000 and β̂₁ = 1,800. What are the coefficients if salary is measured in thousands of dollars? If experience is measured in months?
Show the answer
In thousands: β̂₀ = 42, β̂₁ = 1.8. In months (salary in dollars): β̂₀ = 42,000 is unchanged and β̂₁ = 1,800/12 = $150 per month of experience. R² is the same in every case.
Regression to the mean. Heights of parents and grown children both have mean 68.9 inches and standard deviation 3, with correlation 0.5. Predict the height of a child whose parent is 75 inches tall. Then predict the height of a parent whose child is 75 inches tall.
Show the answer
The slope is r × sd(child)/sd(parent) = 0.5. Child: 68.9 + 0.5 × (75 − 68.9) = 71.95 inches. Parent of a 75-inch child: by symmetry also 71.95 inches. Both predictions regress toward 68.9. Neither line is the “inverse” of the other.
R² is r². Show that in simple regression, SSE = β̂₁² Σ(xi − x̄)², and use it to show that R² equals the squared sample correlation.
Show the answer
ŷi − ȳ = β̂₁(xi − x̄), since the line passes through the means. Squaring and summing gives SSE = β̂₁² Sxx. With β̂₁ = Sxy/Sxx: R² = SSE/SST = Sxy² / (Sxx Syy) = r², where Syy = SST.
Through the origin. Some theories say y = β₁x with no intercept. Derive the least squares slope for that model. When does it equal the usual β̂₁?
Show the answer
Minimize Σ(yi − bxi)². The first-order condition Σxi(yi − bxi) = 0 gives b = Σxiyi / Σxi². It equals the usual slope when x̄ = 0, or when the usual fitted intercept happens to be exactly zero. Otherwise forcing the line through the origin biases the slope if the true intercept isn’t zero, and the residuals no longer need to sum to zero.
How the interactives work
The line. Every time a point moves, the page recomputes x̄, ȳ, Σ(x − x̄)(y − ȳ) and Σ(x − x̄)², then β̂₁, β̂₀, SSR and R², using the formulas above. Each square’s side is the residual measured in screen pixels on the y scale, so all squares share one scale and their total area is proportional to SSR. The bowl is SSR(b) = SST − 2b·Σ(x − x̄)(y − ȳ) + b²·Σ(x − x̄)², the sum of squares of the line with slope b through (x̄, ȳ). The data sets are made up for the lesson.
The heights are simulated. “Parent and child”: both normal with mean 68.9 inches (the average for U.S. men, CDC, 2021–2023) and standard deviation 3 inches (an assumption), correlation 0.5. “Galton’s mid-parents”: mid-parents normal with mean 68.3 and standard deviation 1.8, children = 68.3 + ⅔ × (mid-parent − 68.3) + noise with standard deviation 2.19, which reproduces Galton’s two-thirds and a children’s spread of 2.5 inches. They are not Galton’s actual measurements.
How it was checked (node tests):
- The closed-form slope and intercept match the course library’s matrix OLS, (X′X)⁻¹X′y, to 2 × 10⁻¹² on all five data sets and 500 random ones; residuals sum to zero and are orthogonal to x to 10⁻¹³; SST = SSE + SSR and R² = r² exactly.
- 100,000 random lines near the OLS line: none had a smaller sum of squares. The bowl formula matches the direct sum to 10⁻¹².
- 200,000 simulated parent–child pairs: slope 0.501, reverse slope 0.498, both spreads 3.0; parents from 71 to 73 inches (average 71.9) have children averaging 70.4. Galton’s version: slope 0.668, reverse slope 0.345.
Check yourself
Ordinary least squares chooses the line that minimizes…
Show the answer
Squared vertical residuals. (The plain sum of residuals is zero for any line through the means, so it can’t pick one.)
The OLS slope equals…
Show the answer
β̂₁ = Cov(x, y)/Var(x). It equals the correlation only when x and y have the same standard deviation.
The OLS line always passes through…
Show the answer
From the first-order condition: ȳ = β̂₀ + β̂₁x̄.
A regression has R² = 0.15. This means…
Show the answer
R² is the explained share of variation. A low R² can go with a large, precisely estimated slope.
A company gives a bonus to its 20 lowest-scoring call centers and sees their scores rise next quarter. The best explanation without more data is…
Show the answer
The worst scorers were partly unlucky, and luck doesn’t repeat. A comparison group of low scorers without the bonus is needed.
If you measure y in cents instead of dollars, the OLS slope…
Show the answer
Rescaling y rescales both coefficients by the same factor; R² is unit-free.
Sources
- J. M. Wooldridge, Introductory Econometrics: A Modern Approach, 7th ed. (Cengage, 2020), Chapter 2, sections 2.2 to 2.4 (deriving OLS, its algebraic properties, goodness of fit and units of measurement).
- J. H. Stock and M. W. Watson, Introduction to Econometrics, 4th ed. (Pearson, 2019), Chapter 4.
- OpenStax, Introductory Statistics 2e, 12.3 The Regression Equation.
- Penn State, STAT 501 Regression Methods, Lesson 1: Simple Linear Regression.
- U.S. Bureau of Labor Statistics, Education pays, 2024. CDC, FastStats: Body Measurements.
- F. Galton, “Regression Towards Mediocrity in Hereditary Stature,” Journal of the Anthropological Institute of Great Britain and Ireland 15 (1886): 246–263. J. A. Hanley, “‘Transmuting’ Women into Men: Galton’s Family Data on Human Stature,” The American Statistician 58 (2004), copy at UC Berkeley.
- S. M. Stigler, The History of Statistics: The Measurement of Uncertainty before 1900 (Harvard University Press, 1986), on Legendre, Gauss, Galton and Pearson.