Econometrics · Chapter 11
Panel Data: Fixed and Random Effects
Cities with a lot of crime hire a lot of police. So if you line up cities and compare, more police seem to go with more crime. Follow the same cities over several years, though, and you can ask a sharper question: when a city adds officers, does its own crime fall? Here you’ll watch pooled OLS get the sign wrong, then slide each city’s cloud to the origin and see the true effect appear. You’ll meet first differences, random effects and the Hausman test, and run a Monte Carlo that shows why standard errors must respect the clusters in your data.
View
The panel
Estimates
Missions
What you’re seeing
The panel. Each color is one simulated city, followed for several years. Each city has a hidden baseline level of crime, from local conditions the data don’t record. The true rule is crime = 40 − 0.8 × police + baseline + noise: one more officer per 10,000 residents lowers crime by 0.8 per 1,000 residents. The “hidden link” slider controls how much high-baseline cities hire extra police.
Four views. Raw data shows every city-year and the pooled OLS line, which compares across cities and so mixes the baselines with the police effect. Within subtracts each city’s own averages, sliding every cloud to the origin; only changes inside a city are left, and the slope is the fixed-effects estimate. First differences plots each year’s change against the previous year. City means keeps one point per city: the between estimator, the worst of all here.
The scoreboard. On the right, each estimator’s value and 95% interval against the gold truth. Run 500 panels and the shaded strips show where each estimator lands across samples. The Hausman test at the bottom asks whether random effects can be trusted.
Panel data and unobserved heterogeneity
A panel follows the same N units (people, firms, cities, countries) over T periods. Write the model with a time-constant unobserved effect ai:
Pooled OLS treats ai + uit as one error. If ai is correlated with xit, as when high-crime cities hire more police, pooled OLS suffers heterogeneity bias: omitted-variable bias from Chapter 6, where the omitted variable is everything fixed about the unit. Panel data let us remove ai without ever measuring it.
The key assumption for the fixed-effects and first-difference estimators is strict exogeneity: E(uit | xi1, …, xiT, ai) = 0. Today’s shock can’t be related to the regressors in any period. It rules out, for example, a city hiring police because of a crime spike last year that also shows up in this year’s error.
Fixed effects: the within transformation
Average the equation over time for each unit: ȳi = β₀ + β₁x̄i + ai + ūi. Subtract it from the original. The unobserved effect disappears, because it is the same in every period:
Pooled OLS on these time-demeaned data is the fixed-effects or within estimator. That is exactly what the “Within” view shows: each city’s cloud slides so its average sits at the origin.
Dummy variables give the same answer. Regressing y on x and a dummy for each unit (dropping one, or the intercept) is the least-squares dummy variable (LSDV) regression. Its slope equals the within estimate exactly, and its standard errors come out right automatically. With demeaned data, the degrees of freedom must be corrected to NT − N − k.
What FE can’t do. Any variable that never changes within a unit (a city’s region, a person’s birth year) is wiped out by demeaning, so its effect can’t be estimated. And FE uses only within variation: if x barely changes over time, the estimate is imprecise, and measurement error does more damage, because the true within signal is small next to the noise.
City A goes from 20 to 24 officers per 10,000 and its crime rate from 60.0 to 56.8. City B goes from 30 to 28 officers and its crime from 80.0 to 81.6 (no noise, for clarity).
- Pooled OLS on the four city-years: slope = +2.52. More police, more crime?
- First differences: A changes by (+4, −3.2), B by (−2, +1.6). Slope through the origin: [4 × (−3.2) + (−2) × 1.6] / (4² + 2²) = −16/20 = −0.80.
City B simply has a higher baseline. Comparing each city with itself removes it.
First differences
Another way to remove ai is to subtract last period from this period:
With two periods, first differences and fixed effects give identical estimates and standard errors (when each uses the matching time effects). With more periods they differ. FE is more efficient when the uit are serially uncorrelated; FD is better when the errors follow something close to a random walk. If the two are very different, strict exogeneity may be failing.
Random effects and the Hausman test
Random effects makes a stronger assumption: ai is uncorrelated with the regressors in every period. Then pooled OLS is consistent but inefficient, because the composite error ai + uit is correlated over time within a unit. Generalized least squares fixes this by subtracting only a fraction θ of each unit’s mean:
θ = 0 is pooled OLS; θ = 1 is fixed effects. Random effects can estimate effects of time-constant variables, and is more precise than FE when its assumption holds. In practice θ is estimated from the FE and between regressions.
The Hausman test (Jerry Hausman, 1978) compares the two. If the RE assumption holds, both estimates are consistent and should be close; if it fails, only FE is consistent. With one coefficient:
FE gives −0.81 (se 0.16) and RE −0.19 (se 0.13). H = (−0.81 + 0.19)² / (0.16² − 0.13²) = 0.3844 / 0.0087 = 44.2. The 5% critical value of χ²₁ is 3.84: reject random effects and use fixed effects.
Failing to reject doesn’t prove RE is right; the test may just lack power. Many applied economists use FE by default when the units aren’t a random draw from a larger population (all 50 states, say) or when correlation with ai is plausible.
Two-way fixed effects
Some shocks hit every unit at once: a recession, a national law, a nationwide drop in crime. If those shocks move with x over time (police forces grew while crime fell everywhere), one-way fixed effects mistakes the common trend for the effect of x. Adding a dummy for each period, λt, removes everything common to all units in a year:
In the simulation, tick “Nationwide trend”: police rise 0.6 a year and crime falls 1.5 a year in every city for reasons unrelated to police. One-way FE drifts to about −1.4. Tick “Year effects” and it returns to −0.8. Two-way fixed effects is also the workhorse behind difference-in-differences, the subject of Chapter 12, which also explains when it can mislead.
It removes only what is fixed over time within each unit. A confounder that changes over time (a new mayor who both hires police and launches other programs) survives. So does reverse causality from this year’s crime to this year’s hiring, and measurement error gets worse, not better. Fixed effects is a strong tool for one specific problem.
Clustered standard errors
Observations from the same city, school or firm tend to share shocks, so they carry less independent information than their count suggests. If the regressor is also similar within groups, ordinary and even heteroskedasticity-robust standard errors are too small. Brent Moulton (1990) showed how much: with groups of size m, the variance is understated by roughly the Moulton factor.
Clusters
Goals
The fix. Cluster-robust standard errors (Kung-Yee Liang and Scott Zeger, 1986; Manuel Arellano, 1987 for panels) add up residual-weighted regressors within each cluster before squaring, so any correlation inside a cluster is allowed. They need many clusters, not many observations: with few clusters (under 20 to 50, depending on the setting) they are too small too, and methods like the wild cluster bootstrap help. The simulation’s clustered intervals use t critical values with G − 1 degrees of freedom, a common small-sample adjustment.
Students in 50 schools, 20 per school. A policy is set by school, so its within-school correlation is 1, and test-score errors have a within-school correlation of 0.2. The Moulton factor is 1 + 19 × 1 × 0.2 = 4.8, so classic standard errors are √4.8 ≈ 2.2 times too small. A t statistic of 4 is really about 1.8: not significant at 5%.
Which level? Cluster at the level where shocks are shared and where the regressor varies: if a policy changes by state, cluster by state even if the data are on people or counties. In panels, cluster by unit, which also handles serial correlation within each unit over time. Marianne Bertrand, Esther Duflo and Sendhil Mullainathan (2004) showed that ignoring serial correlation in state panels made placebo “laws” look significant far too often.
Colin Cameron and Douglas Miller’s 2015 practitioner’s guide is the standard reference.
A short history
Fixed effects. Yair Mundlak (1961) used farm panels to estimate production functions free of “management bias”: each farm’s unobserved management skill was a fixed effect, removed by comparing a farm with itself. In 1978 he showed that random effects with the unit means of the regressors added reproduces fixed effects, the “Mundlak device.”
Random effects. Pietro Balestra and Marc Nerlove (1966) developed the GLS random-effects estimator to study the demand for natural gas across U.S. states. Jerry Hausman’s 1978 Econometrica paper on specification tests gave the test that bears his name.
Clustering. Moulton’s 1990 note showed how badly ignoring group structure can mislead, and the 2004 paper by Bertrand, Duflo and Mullainathan made clustering standard practice in policy evaluation.
Where it shows up
Firms and managers. Firm fixed effects absorb management quality, culture and location. Studies of what a policy or investment does to productivity typically compare each firm or plant with itself before and after. Economists studying Henry Ford’s 1914 jump to a five-dollar day compared the same company’s turnover and absences before and after (see the Ford case).
Workers. Wage panels follow the same people as they join or leave unions, move cities or change jobs; person fixed effects remove fixed ability, the problem from Chapter 6.
Marketing and sports. Store fixed effects control for neighborhood and size when measuring promotions; player and team fixed effects separate talent from circumstance in sports analytics.
Policy. State-year panels with state and year fixed effects, clustered by state, are the standard design for studying minimum wages, taxes and regulations.
Practice
Warm-up A city had police levels 20, 22 and 27 over three years. What are the time-demeaned values?
Show the answer
The mean is 23, so −3, −1 and +4. They sum to zero, as demeaned values always do.
Warm-up You run a fixed-effects regression of wages on hours trained, union status and gender, with worker fixed effects. Which coefficient can’t be estimated, and why?
Show the answer
Gender, which doesn’t change over time for a worker. Demeaning wipes it out; its effect is absorbed into the worker fixed effect.
Core Store A’s sales went from 100 to 112 when its ad budget went from 10 to 14; store B’s from 200 to 197 when its budget went from 30 to 29. Estimate the ad effect by first differences (no intercept).
Show the answer
Δ’s: A (4, 12), B (−1, −3). β̂ = (4 × 12 + (−1)(−3)) / (16 + 1) = 51/17 = 3.0 per unit of ad budget.
Core With σ²u = 4, σ²a = 36 and T = 6, compute the random-effects θ. Is RE closer to pooled OLS or to FE?
Show the answer
θ = 1 − √[4/(4 + 6 × 36)] = 1 − √(4/220) = 1 − 0.135 = 0.865: close to FE (θ = 1). Large unit effects relative to the noise push RE toward FE.
Core FE gives 0.42 (se 0.10) and RE gives 0.35 (se 0.08). Compute the Hausman statistic and decide at 5%.
Show the answer
H = 0.07² / (0.0100 − 0.0064) = 0.0049 / 0.0036 = 1.36 < 3.84. Don’t reject; RE is not contradicted by the data (which isn’t proof that it is right).
Core A study has 40 classrooms of 25 students; the treatment is assigned by classroom (ρx = 1), and the error’s within-classroom correlation is 0.1. Compute the Moulton factor and the factor by which classic standard errors are too small.
Show the answer
1 + 24 × 1 × 0.1 = 3.4, so the standard errors are √3.4 ≈ 1.84 times too small. Even a modest within-class correlation matters when the treatment is at the class level.
Challenge Show that with T = 2, the within estimator equals the first-difference estimator (no intercept).
Show the answer
With two periods, xi1 − x̄i = −Δxi/2 and xi2 − x̄i = Δxi/2, and the same for y. So Σi,t ẍitÿit = Σi 2 × (Δxi/2)(Δyi/2) = ½ΣΔxiΔyi, and Σẍ² = ½ΣΔxi². The halves cancel: β̂FE = ΣΔxΔy/ΣΔx² = β̂FD.
How the interactive works
The panel. Each city gets a baseline ai, normal with the chosen spread. Police = 25 + 3κ(ai/σa) + 2ξit, so κ (the hidden link) sets how much baselines drive hiring; crime = 40 − 0.8 × police + ai + 2uit. The nationwide trend adds 0.6 officers and −1.5 crimes per year everywhere. Simulated with a known truth; the numbers are illustrative, not estimates for real cities. In large samples pooled OLS heads to −0.8 + 3κσa/(9κ² + 4).
The estimators. Fixed effects is the dummy-variable regression from the shared statistics library (the within estimate is shown alongside); first differences regress changes on changes; random effects quasi-demeans with Swamy–Arora variance components and uses the GLS variance; the Hausman test uses the classic variances. The clustering simulation fits y = 1 + 0.5x + u, where x and u each share a cluster component, and computes classic, HC1 and cluster-robust standard errors.
How it was checked (node tests, all passed):
- The within slope equaled the dummy-variable slope (one- and two-way) to 5 × 10⁻¹³; with two years, FE equaled FD in coefficient and standard error to 3 × 10⁻¹³. Random effects matched full GLS in NumPy (coefficient and standard error) to 10⁻¹⁴.
- 1,000 panels of 8 cities × 6 years with κ = 0.9: FE averaged −0.813, FD −0.810, RE −0.189, pooled +0.500, between +1.233; Hausman rejected in 99.5%. With 300 cities pooled OLS averaged 0.633 against the formula’s 0.635. With a nationwide trend, one-way FE averaged −1.413 and two-way FE −0.815.
- With κ = 0 and 30 cities, the Hausman test rejected in 5.8% of 2,000 panels (nominal 5%).
- Classic, HC1 and clustered standard errors matched the shared library to 4 × 10⁻¹⁵. Over 4,000 samples of 50 clusters × 20 with both correlations 0.5: classic coverage 58.9%, robust 58.6%, clustered 93.8%; the spread was 2.37 times the classic standard error against √Moulton = 2.40. With 10 clusters, clustered coverage fell to 91.7%.
Check yourself
Pooled OLS across cities shows more police with more crime. Fixed effects shows the opposite. The most likely explanation is…
Show the answer
The unobserved baseline is correlated with police. FE removes it by comparing each city with itself.
The within transformation subtracts…
Show the answer
Subtracting last period’s value is first differencing; subtracting the unit’s average is the within (FE) transformation.
Random effects is consistent only if…
Show the answer
That is the RE assumption; the Hausman test checks whether the data contradict it.
Police forces grew and crime fell in every city over the decade for unrelated reasons. One-way FE will…
Show the answer
Within each city, police and crime both trend, so one-way FE picks up the trend. Year dummies absorb shocks common to all cities.
A state policy is studied with data on 100,000 workers in 50 states. Standard errors should be clustered by…
Show the answer
The policy varies by state and workers in a state share shocks, so the effective sample is closer to 50 than to 100,000.
Sources
- J. M. Wooldridge, Introductory Econometrics: A Modern Approach, 7th ed. (Cengage, 2020), Chapter 13 (Pooling Cross Sections across Time: Simple Panel Data Methods) and Chapter 14 (Advanced Panel Data Methods): first differences, fixed effects, random effects, the Hausman test and clustering in panels.
- J. H. Stock and M. W. Watson, Introduction to Econometrics, 4th ed. (Pearson, 2019), Chapter 10 (Regression with Panel Data), including a traffic-fatalities panel and clustered standard errors.
- J. D. Angrist and J.-S. Pischke, Mostly Harmless Econometrics (Princeton University Press, 2009), Chapter 5 (Parallel Worlds: Fixed Effects, Differences-in-Differences, and Panel Data) and Section 8.2 (clustering and serial correlation).
- MIT OpenCourseWare, 14.32 Econometrics (Spring 2007).
- J. A. Hausman, “Specification Tests in Econometrics,” Econometrica 46(6), 1978, 1251–1271. Y. Mundlak, “On the Pooling of Time Series and Cross Section Data,” Econometrica 46(1), 1978, 69–85. P. Balestra and M. Nerlove, “Pooling Cross Section and Time Series Data in the Estimation of a Dynamic Model: The Demand for Natural Gas,” Econometrica 34(3), 1966, 585–612.
- B. R. Moulton, “An Illustration of a Pitfall in Estimating the Effects of Aggregate Variables on Micro Units,” Review of Economics and Statistics 72(2), 1990, 334–338. M. Bertrand, E. Duflo and S. Mullainathan, “How Much Should We Trust Differences-in-Differences Estimates?” Quarterly Journal of Economics 119(1), 2004, 249–275.
- A. C. Cameron and D. L. Miller, “A Practitioner’s Guide to Cluster-Robust Inference,” Journal of Human Resources 50(2), 2015, 317–372.