Econometrics · Chapter 17

Modern Causal Inference

“Control for everything” sounds safe. It isn’t. Some controls remove bias, some add it, and a few quietly change the question you’re answering. In this last chapter you’ll draw causal diagrams and let the page work out which variables to adjust for, then watch a simulation prove it right and catch the traps. You’ll match treated and untreated people on their propensity scores, build a synthetic twin for a state that changed its policy, and see why machine learning is great at prediction but needs a careful recipe before it can answer a causal question.

Diagram

Clicking a node

Control for

Add, flip or remove an arrow

Live numbers

Missions

    What you’re seeing

    The diagram. Each circle is a variable; an arrow from A to B means A directly causes B. D is the treatment (a training program), Y the outcome (wages). Dashed circles are variables nobody measured. The page lists every path from D to Y, following arrows in either direction, and colors it: green paths carry the causal effect; red paths are open back doors that mix in other influences; grey paths are blocked.

    The simulation. Behind the diagram, the page creates 2,000 people whose variables follow it exactly: each variable is 0.8 times each of its causes (1.0 for D → Y), plus noise. So the true effect of D on Y is known. It then runs the regression you chose, Y on D plus whatever you control for, and plots the estimate against the truth. “Bias in a huge sample” computes what that regression would give with infinite data.

    Your moves. Click a node to control for it (put it in the regression), or switch to drawing arrows and click two nodes to connect them. The page applies the backdoor criterion and lists every set of observed variables that would give an unbiased answer.

    Potential outcomes and causal diagrams

    Two languages describe cause and effect, and modern econometrics uses both. The potential outcomes framework (Jerzy Neyman, 1923; Donald Rubin, 1974) gives every person two outcomes: Y(1) with the treatment and Y(0) without. Only one is ever seen. Comparing averages of the treated and untreated mixes the effect with selection:

    E[Y | D=1] − E[Y | D=0] = ATT + (E[Y(0) | D=1] − E[Y(0) | D=0])naive comparison = average effect on the treated + selection bias (how the treated would have differed anyway)

    Causal diagrams (directed acyclic graphs, or DAGs), championed by Judea Pearl, show why there is selection bias and what would remove it. Every path between D and Y is built from three kinds of junctions:

    Chain: A → B → C

    B passes A’s effect on to C. Controlling for B blocks the flow. If D → M → Y, controlling for M removes part of the effect you want.

    Fork: A ← B → C

    B causes both, so A and C move together even if neither causes the other. Controlling for B blocks the spurious link. This is a confounder.

    Collider: A → B ← C

    A and C both cause B. The path is blocked as long as you leave B (and everything B causes) alone. Controlling for B opens it and creates a link between A and C that isn’t causal.

    A back-door path is a path from D to Y that starts with an arrow pointing into D. Those paths carry association that isn’t D’s effect. Pearl’s criterion tells you what to control for:

    Backdoor criterion: a set Z identifies the effect of D on Y if (1) no variable in Z is caused by D, and (2) Z blocks every back-door paththen comparing people with the same Z (for example, regressing Y on D and Z in a linear model) gives the causal effect
    Worked example: how big the confounding bias is

    In the “Confounder” diagram, motivation X raises training (D = 0.8X + noise) and wages (Y = 1.0·D + 0.8X + noise), all noises with variance 1.

    1. Var(D) = 0.8² + 1 = 1.64, and Cov(X, D) = 0.8.
    2. Regressing Y on D alone gives 1.0 + 0.8 × Cov(X, D)/Var(D) = 1.0 + 0.8 × 0.8/1.64 = 1.39. That’s the omitted-variable bias formula from Chapter 6: 39% too high.
    3. Controlling for X blocks D ← X → Y, the only back door, and the coefficient on D is exactly 1.0.

    The simulation reproduces both numbers: 1.391 and 0.998 with 200,000 simulated people.

    Try this: open “Mediator” and control for skills. The estimate falls from 1.14 to 0.50: you’ve measured only the part of training’s effect that doesn’t go through skills. Then open “Collider” and control for promotion: the estimate drops from 1.00 to 0.22, though nothing about training changed. Finally, open “M-bias”: the survey answer C is measured before training and looks like a harmless control, yet controlling for it opens D ← U1 → C ← U2 → Y and biases the estimate (0.88 instead of 1.00).
    Common mistake: “Control for every variable you have”

    Adding controls is safe only if they are not caused by the treatment (no mediators and none of their consequences) and don’t open new paths (no colliders). Angrist and Pischke call variables that are themselves outcomes of the treatment “bad controls”. A classic example of collider bias is Berkson’s paradox: among hospital patients, two unrelated diseases can look negatively related, because having either one is a reason to be in the hospital. The fix is not a rule of thumb but a diagram: write down what causes what, then check the paths.

    And when no observed set works? Choose “Hidden confounder”: ability (unobserved) drives both training and wages, so no regression on observables is unbiased, and controlling for the lottery Z makes it worse (1.39 instead of 1.28, a known effect called bias amplification). But the lottery is an instrument: it moves training and affects wages only through training. The page reports the IV estimate, which lands on the truth.

    Selection on observables: matching and weighting

    If the back doors all run through variables you observe, two assumptions are enough. Unconfoundedness: given the covariates X, treatment is as good as random, (Y(0), Y(1)) ⊥ D | X. Overlap: everyone has some chance of each status, 0 < P(D = 1 | X) < 1. Then compare treated and untreated people with the same X. With many covariates, exact matches are rare, but Paul Rosenbaum and Donald Rubin (1983) proved you only need to match on one number, the propensity score p(X) = P(D = 1 | X): if treatment is unconfounded given X, it is unconfounded given p(X).

    In this simulation, two covariates (think age and prior earnings) make people more likely to join a program, and also raise their earnings anyway. The program’s effect grows with the first covariate, so the average effect on the treated (ATT) is larger than the average effect for everyone (ATE). Each method answers a specific question: matching each treated person to the untreated person with the closest score estimates the ATT; inverse probability weighting can target either; a regression with the covariates gives a weighted average of effects that is neither, when effects vary.

    IPW: ATE = E[D·Y/p(X)] − E[(1 − D)·Y/(1 − p(X))],   with the weights normalized to sum to 1 in each group
    Worked example: weighting by hand

    Eight people. Four have X = 1 and a 75% chance of treatment: three treated (earning 10, 12 and 11) and one not (8). Four have X = 0 and a 25% chance: one treated (6) and three not (4, 5 and 3).

    1. Naive: treated average (10 + 12 + 11 + 6)/4 = 9.75; untreated (8 + 4 + 5 + 3)/4 = 5.0; difference 4.75.
    2. Within groups: X = 1: 11 − 8 = 3; X = 0: 6 − 4 = 2. The groups are equal in size, so the ATE is 2.5.
    3. Weights: treated people get 1/p: 1.33 each for X = 1, 4 for X = 0. Weighted treated mean = (1.33 × 33 + 4 × 6)/(1.33 × 3 + 4) = 68/8 = 8.5. Untreated get 1/(1 − p): 4 for X = 1, 1.33 each for X = 0: (4 × 8 + 1.33 × 12)/8 = 48/8 = 6.0.
    4. IPW estimate: 8.5 − 6.0 = 2.5, the same as comparing within groups.

    The rare cases (the one treated person with X = 0) get large weights. If a score is near 0 or 1, one person can dominate the estimate: that is what poor overlap looks like.

    Try this: raise the selection strength to 1: matching balances both covariates (standardized differences fall from about 0.9 to under 0.1) and lands near the true ATT. Now push it to 3. The treated pile up near a score of 1, the untreated near 0, and few people overlap. Some untreated people get IPW weights near 100, the estimates wobble, and balance after matching gets worse. Tick the caliper: matching now drops treated people with no close control, which improves balance but changes whom the estimate describes. No method can create overlap that isn’t in the data.

    A famous test of these methods: Robert LaLonde (1986) took a randomized job-training experiment, the National Supported Work Demonstration, replaced its control group with people from general surveys, and showed that many standard observational estimators failed to reproduce the experimental answer. Rajeev Dehejia and Sadek Wahba (1999) found that propensity-score methods came much closer on a subsample, a result later debated by Jeffrey Smith and Petra Todd (2005). The lesson both sides agree on: these methods are only as good as the covariates and the overlap.

    Synthetic control: building a twin

    When one state or one company is treated, there may be no single good comparison. Alberto Abadie and Javier Gardeazabal (2003) and Abadie, Alexis Diamond and Jens Hainmueller (2010) proposed building one: a weighted average of untreated units, with weights that are non-negative and add to 1, chosen so that the average tracks the treated unit before the policy.

    minw Σt < T₀ (Y1t − Σj wjYjt)²,   wj ≥ 0,   Σwj = 1after T₀, the gap between the treated unit and its synthetic twin estimates the effect

    True effect of the policy

    Inference by placebos. With one treated unit, there are no standard errors in the usual sense. Instead, pretend each donor was treated, build its own twin from the others, and see how big its post-policy gap is relative to its pre-policy fit. If the real treated unit’s ratio is the largest of 16, the chance of that happening by luck is 1/16 ≈ 0.06.

    Worked example: a twin by hand

    The weights put 0.5 on state A, 0.3 on B and 0.2 on C. In the year before the policy, A, B and C had 120, 90 and 100 packs of cigarettes sold per person, and the treated state had 107.5.

    1. Synthetic value before: 0.5 × 120 + 0.3 × 90 + 0.2 × 100 = 60 + 27 + 20 = 107, close to 107.5.
    2. Three years later, A, B and C sold 110, 85 and 95: synthetic = 55 + 25.5 + 19 = 99.5.
    3. The treated state sold 92. Estimated effect: 92 − 99.5 = −7.5 packs per person.

    The original application studied California’s Proposition 99, a 1988 tobacco-control program that raised the cigarette tax by 25 cents a pack and funded anti-smoking efforts. Abadie, Diamond and Hainmueller built a synthetic California from other states and estimated that by 2000, annual cigarette sales in California were about 26 packs per person lower than they would have been without the program.

    Try this: with equal weights, the twin is just the donors’ average and misses the treated state badly before the policy. Drag a few weight bars and watch the fit change, then press “Fit the weights”. The fit before the policy gets very close, but the weights the optimizer picks are not the ones the simulation used to build the treated state (0.5, 0.3, 0.2 on units 2, 5 and 9): with 20 years and a few underlying trends, many different blends fit equally well. What matters is the fit, and the placebo test.

    Machine learning: prediction vs causation

    Machine learning is built to predict: given x, guess y as well as possible on new data. Causal questions are different: what happens to y if we change d? A model can predict brilliantly using variables that are consequences of y, or proxies for d, and be useless for that question. But ML can still help causal inference, if it is used only for the parts of the problem that are pure prediction.

    The LASSO (Robert Tibshirani, 1996) fits a regression while penalizing the size of the coefficients, which sets many of them exactly to zero:

    minb (1/2n) Σ(yi − xi′b)² + λ Σj |bj|

    With λ = 0 it is OLS; as λ grows, coefficients shrink and drop out one by one. With 51 variables and only 100 observations, OLS nearly memorizes the training data (in our test, R² 0.96 there but 0.85 on new data); a moderate λ keeps about 15 variables and predicts new data better (0.93).

    Why naive selection fails, and how double/debiased ML fixes it

    Suppose we want the effect θ of a treatment d on y, with 50 possible controls. A tempting recipe: let LASSO pick the controls that predict y, then run OLS of y on d and those controls. It fails when a confounder strongly drives d but only mildly drives y: LASSO drops it as unimportant for predicting y, and its omitted-variable bias comes back. In the simulation, three such confounders push the naive estimate to about 1.23 for a true θ of 1, and its 95% intervals almost never contain the truth.

    Double/debiased machine learning (Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey and James Robins, 2018) builds on a partialling-out idea (Peter Robinson, 1988) in three steps:

    1. Predict y from the controls with any ML method, and keep the residual ũ = y − ℓ̂(x).
    2. Predict d from the controls, and keep the residual ṽ = d − m̂(x). This step catches the confounders that drive d.
    3. Regress ũ on ṽ: θ̂ = Σṽiũi / Σṽi². Use cross-fitting: predictions for each observation come from a model trained on the other part of the sample, so overfitting can’t leak into the estimate.

    The estimating equation is Neyman-orthogonal: small errors in the two predictions have no first-order effect on θ̂, so θ̂ is approximately normal with a usual standard error even though the predictions are imperfect. In our simulations with 400 observations, it averaged 0.995 and its intervals covered the truth 94% of the time. Two cautions, stated carefully: DML still needs unconfoundedness (all confounders among the controls), and the predictions must be reasonably good; it fixes the estimation problem, not the identification problem. (In our tests, using plain LASSO predictions instead of refitting the selected variables by OLS left visible shrinkage bias: coverage 73%.)

    Worked example: partialling out by hand

    For five people, the residuals from predicting y are ũ = (2, −1, 0, 1, −2) and from predicting d are ṽ = (1, −1, 0.5, 0, −0.5).

    1. Σṽũ = 2 + 1 + 0 + 0 + 1 = 4.
    2. Σṽ² = 1 + 1 + 0.25 + 0 + 0.25 = 2.5.
    3. θ̂ = 4/2.5 = 1.6.

    This is the Frisch–Waugh–Lovell theorem from Chapter 6, with machine-learned predictions in place of linear regressions.

    Where the field is going

    • Design first. The “credibility revolution” put the research design (experiments, natural experiments, discontinuities, clean comparisons) ahead of the regression. David Card, Joshua Angrist and Guido Imbens shared the 2021 Nobel memorial prize in economics for this work.
    • Heterogeneous effects. Methods such as causal forests (Stefan Wager and Susan Athey, 2018) estimate how effects vary across people, for targeting a program or a discount to those it helps most.
    • Combining designs. Synthetic difference-in-differences (Dmitry Arkhangelsky and coauthors, 2021) blends the weights of synthetic control with the logic of Chapter 12.
    • Honest uncertainty. Sensitivity analysis asks how strong an unmeasured confounder would have to be to overturn a result, instead of assuming there is none.
    • Experiments at scale. Tech companies run thousands of randomized A/B tests a year, and the questions in this course (interference between users, long-run effects, heterogeneous effects) are now everyday business problems.

    A short history

    Path diagrams. The geneticist Sewall Wright drew arrows between variables and multiplied coefficients along paths in the 1920s, an ancestor of both DAGs and the simulation on this page.

    Potential outcomes. Jerzy Neyman wrote them down for agricultural experiments in 1923; Donald Rubin extended them to observational studies in the 1970s. Rosenbaum and Rubin’s propensity score followed in 1983.

    Graphs. Judea Pearl developed the graphical theory of causation, including the backdoor criterion, in the 1990s; his book Causality appeared in 2000. He received the 2011 Turing Award, in part for this work.

    New tools. Synthetic control (2003, 2010), the LASSO (1996), double selection (Alexandre Belloni, Chernozhukov and Hansen, 2014) and double/debiased machine learning (2018) brought the field to where it is today.

    Where it shows up

    • Business. Did the loyalty program raise spending, or did loyal customers join it? Propensity scores and DAGs make that question precise. When a company changes policy in one market, synthetic control builds a comparison from the others. The “Don’t buy this jacket” campaign (Patagonia) is the kind of one-time move whose effect on sales you would want to compare with a synthetic twin rather than with last year.
    • Policy. Tobacco taxes, minimum wages and immigration shocks in single states or cities are classic synthetic-control applications.
    • Medicine. Observational studies of drugs use propensity scores and DAGs to choose adjustments; collider bias explains several famous paradoxes, such as low-birth-weight babies of smoking mothers appearing to do better than other low-birth-weight babies.
    • Technology. Double machine learning is used to estimate price elasticities and advertising effects with thousands of possible controls.
    • Daily life. “Students who hire tutors get lower grades, so tutoring hurts” ignores a fork: struggling students hire tutors. Draw the diagram before you draw the conclusion.

    Practice

    1. In the diagram A → D → Y, A → Y, what should you control for to estimate the effect of D on Y?

      Show the answer

      A. The path D ← A → Y is a back door (it starts with an arrow into D), and A is a fork on it; controlling for A blocks it. A is not caused by D, so the backdoor criterion holds.

    2. D → M → Y and D → Y. A colleague controls for M. What does the coefficient on D now measure?

      Show the answer

      The direct effect of D on Y, not the total effect: controlling for the mediator blocks the path through M. If the question is “what does D do to Y in total?”, don’t control for M.

    3. Talent and looks are independent in the population. A movie studio only casts actors who are talented or good-looking (or both). Among cast actors, why might talent and looks be negatively related?

      Show the answer

      Casting is a collider: Talent → Cast ← Looks. Looking only at cast actors conditions on it. An actor who isn’t good-looking must be talented to be cast, and vice versa, which creates a negative association among the cast that doesn’t exist in the population. That’s Berkson’s paradox.

    4. Two people have propensity scores of 0.9 (treated) and 0.1 (untreated). What IPW weights do they get, and what does a score of 0.99 for an untreated person imply?

      Show the answer

      Treated: 1/0.9 ≈ 1.11. Untreated with 0.1: 1/(1 − 0.1) ≈ 1.11. An untreated person with a score of 0.99 gets 1/0.01 = 100: they stand in for 100 similar people who were treated, so the estimate depends heavily on one person. That signals poor overlap; consider trimming or a different target.

    5. A synthetic control gives the treated state a post-policy gap of −8 and pre-policy RMSPE of 2. Its post/pre RMSPE ratio ranks 3rd of 40 placebos. What is the placebo p-value?

      Show the answer

      3/40 = 0.075. Two placebo states had larger ratios by chance, so the evidence is suggestive but not strong at the 5% level.

    6. Residuals from predicting y and d with ML give Σṽũ = 12 and Σṽ² = 30. Estimate θ. Why must the predictions be cross-fitted?

      Show the answer

      θ̂ = 12/30 = 0.4. If the same data trained the prediction models and computed the residuals, a flexible model could fit the noise, making the residuals too small and correlated with each other, which biases θ̂. Cross-fitting uses predictions from models that never saw that observation.

    7. An analyst says: “I used a neural network with 500 variables, so confounding isn’t a problem.” What is wrong?

      Show the answer

      Flexible prediction doesn’t identify causal effects. If a confounder isn’t among the 500 variables, no algorithm can adjust for it; if some of the 500 are mediators or colliders, adjusting for them adds bias. The design (which variables, which comparison) still decides whether the answer is causal.

    How the interactive works

    Diagrams. The page lists every path between D and Y in the diagram, ignoring direction, and checks each junction: a path is blocked by a controlled chain or fork node, or by a collider that is not controlled and has no controlled descendant (Pearl’s d-separation). The backdoor criterion and all minimal valid adjustment sets are found by checking every subset of the observed variables. The simulation is a linear model in the diagram’s order with standard normal noise; “bias in a huge sample” uses the model’s exact covariance matrix, (I − B)−1(I − B)−′.

    Matching. The propensity score is a logit fitted with the course library; matching is one nearest neighbor on the score, with replacement; IPW is normalized. Synthetic control. Weights minimize the pre-period squared gap on the simplex by accelerated projected gradient, after removing each year’s cross-unit average (which doesn’t change the answer when weights add to 1, and makes it converge much faster). ML. LASSO by coordinate descent on standardized variables; the penalty for the causal estimators is 1.1 σ̂ √(2 ln p/n); DML uses 2-fold cross-fitting with post-LASSO predictions.

    How it was checked (Node.js, NumPy, SciPy and scikit-learn, before the page was built):

    • With 200,000 simulated people, the regression matched the exact large-sample values in every diagram: confounder 1.391 / 0.998 (with X), mediator 1.141 / 0.504 (with M), collider 1.000 / 0.220 (with C), M-bias 1.000 / 0.877 (with C), hidden confounder 1.280 / 1.391 (with Z). The IV estimate was 0.995 (truth 1) and matched the shared library’s 2SLS to six decimals on a smaller sample.
    • Matching, 200 samples of 1,000 at selection strength 1: true ATT 1.359 and ATE 0.998; matching 1.373, IPW-ATT 1.362, IPW-ATE 1.020, regression 1.145, naive 2.943; median largest |SMD| after matching 0.088. At strength 3, the largest IPW weight had a median of 88 and IPW-ATE averaged 1.36.
    • Synthetic control: the weights matched SciPy’s constrained optimizer to 0.0015 with the same pre-period RMSPE (0.30430), and the optimality (KKT) conditions held. The treated unit’s post/pre ratio (13.1) was the largest of 16.
    • LASSO coefficients matched scikit-learn to 10−10. DML, 300 studies with 400 observations: mean 0.995, SD 0.053, coverage 94.0%; naive selection: mean 1.228, coverage 0%. With 200 observations: DML 1.000, coverage 90%.

    Check yourself

    1. Which variable should you NOT control for when estimating the total effect of D on Y?

      Show the answer

      That’s a mediator. Controlling for it blocks part of the effect you want. A confounder should be controlled for; a variable that only causes Y is harmless and can improve precision.

    2. What happens when you control for a collider on a path between D and Y?

      Show the answer

      A collider blocks its path on its own. Conditioning on it (or on its consequences) opens the path.

    3. The propensity score theorem says that if treatment is unconfounded given X, then…

      Show the answer

      Rosenbaum and Rubin (1983): comparing treated and untreated people with the same propensity score is enough, which turns many covariates into one.

    4. How is a synthetic control’s estimate judged, since there is only one treated unit?

      Show the answer

      If the treated unit’s gap (relative to its pre-period fit) is extreme compared with the gaps of units that had no policy, the effect is unlikely to be luck.

    5. Why does “use LASSO to pick the controls that predict y, then run OLS” fail?

      Show the answer

      Prediction of y doesn’t prioritize what matters for confounding. Double selection or DML also models d, so those confounders are caught.

    Sources

    • J. Pearl, Causality: Models, Reasoning, and Inference, 2nd ed. (Cambridge University Press, 2009), chapter 3 (the backdoor criterion); J. Pearl, M. Glymour and N. P. Jewell, Causal Inference in Statistics: A Primer (Wiley, 2016).
    • G. W. Imbens and D. B. Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences (Cambridge University Press, 2015).
    • J. D. Angrist and J.-S. Pischke, Mostly Harmless Econometrics (Princeton, 2009), chapter 3 (regression, matching, propensity scores and bad controls). S. Cunningham, Causal Inference: The Mixtape (Yale University Press, 2021), chapters on DAGs, matching and synthetic control.
    • Stanford Encyclopedia of Philosophy, Causal Models.
    • P. R. Rosenbaum and D. B. Rubin, The central role of the propensity score in observational studies for causal effects, Biometrika 70(1), 41–55 (1983). R. J. LaLonde, Evaluating the econometric evaluations of training programs with experimental data, American Economic Review 76(4), 604–620 (1986). R. H. Dehejia and S. Wahba, Causal effects in nonexperimental studies, Journal of the American Statistical Association 94(448), 1053–1062 (1999). J. A. Smith and P. E. Todd, Does matching overcome LaLonde’s critique of nonexperimental estimators?, Journal of Econometrics 125(1–2), 305–353 (2005).
    • A. Abadie and J. Gardeazabal, The economic costs of conflict: a case study of the Basque Country, American Economic Review 93(1), 113–132 (2003). A. Abadie, A. Diamond and J. Hainmueller, Synthetic control methods for comparative case studies: estimating the effect of California’s tobacco control program, Journal of the American Statistical Association 105(490), 493–505 (2010).
    • R. Tibshirani, Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society B 58(1), 267–288 (1996). A. Belloni, V. Chernozhukov and C. Hansen, Inference on treatment effects after selection among high-dimensional controls, Review of Economic Studies 81(2), 608–650 (2014). V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey and J. Robins, Double/debiased machine learning for treatment and structural parameters, Econometrics Journal 21(1), C1–C68 (2018).
    • S. Wager and S. Athey, Estimation and inference of heterogeneous treatment effects using random forests, Journal of the American Statistical Association 113(523), 1228–1242 (2018). D. Arkhangelsky, S. Athey, D. A. Hirshberg, G. W. Imbens and S. Wager, Synthetic difference-in-differences, American Economic Review 111(12), 4088–4118 (2021).
    • Wikipedia (secondary): Rubin causal model, Collider (statistics), Berkson’s paradox, Synthetic control method, Lasso (statistics).
    Saved on this device only.
    @teddyrookbook GitHub