Econometrics · Chapter 14
Binary Outcomes: Logit, Probit and Maximum Likelihood
Did the customer buy? Was the loan approved? Did the student pass? Many of the outcomes businesses care about are a plain yes or no. Fit an ordinary regression line to them and it soon predicts a 130% chance of buying, or a −20% chance of passing. Here you’ll put the straight line next to two S-shaped curves, logit and probit, drag the data around to see which one bends and which one breaks, and then climb a mountain of likelihood by hand and with Newton’s method to see how the S-curves are actually fitted.
The story
The true model
Click the plot to add a person: above the dotted middle line for a yes, below for a no. Drag a dot to move it or flip it; double-click it (or drag it off the plot) to remove it.
Live numbers
Missions
What you’re seeing
The data. Each dot is one simulated person. Yes answers sit at the top (1), no answers at the bottom (0); they’re spread a little up and down only so you can see them. The white circles show the share of yeses among people with similar x: that’s the probability we’re trying to trace.
The straight line. The violet line is ordinary least squares on the 0s and 1s, called the linear probability model. It’s easy to read (the slope is the change in probability per unit of x) but it doesn’t know that probabilities stop at 0 and 1, so at the ends it wanders into the red zones.
The S-curves. The green logit and the dashed gold probit curves are fitted by maximum likelihood. They bend to stay between 0 and 1, and are almost indistinguishable from each other in the middle. Their coefficients are on a different scale, so the table translates them into partial effects: how much the probability changes per unit of x.
Play with it. Add a surprising yes far to the left and watch which model reacts most. Pull points apart until every yes is to the right of every no, and the S-curves have no finite answer at all: that’s called perfect separation.
The straight line and its trouble
With a 0/1 outcome, the population regression is a probability: E[y | x] = P(y = 1 | x). The linear probability model (LPM) sets it to a line, P(y = 1 | x) = β₀ + β₁x, and estimates it by OLS. Wooldridge (section 7.5) points out what is good and bad about it:
- Easy to read. β₁ is the change in the probability for a one-unit change in x. A slope of 0.09 on hours of practice means each hour adds about 9 percentage points to the chance of passing.
- Impossible predictions. A line has to leave [0, 1] somewhere, often inside the range of the data, as the red bands show.
- Constant effects. The line says the 1st hour of practice and the 12th each add the same 9 points, which can’t be true when the chance is already near 100%.
- Built-in heteroskedasticity. For a 0/1 outcome, Var(y | x) = p(x)(1 − p(x)), which changes with x. So always use robust standard errors with an LPM (see Chapter 8).
In practice the LPM often gives average effects close to the logit’s and probit’s, especially when most probabilities are between 0.2 and 0.8. Its problems bite at the extremes.
The latent-variable model
Imagine each person has a hidden score, y* = β₀ + β₁x + e: a customer’s net desire for the upgrade, a borrower’s creditworthiness. We only see whether it crosses zero: y = 1 if y* > 0. If e is symmetric around zero with distribution function G, then
e has the logistic distribution: G(z) = Λ(z) = exp(z)/(1 + exp(z)). Its log-odds are linear: ln[p/(1 − p)] = β₀ + β₁x.
e is standard normal: G(z) = Φ(z), the normal distribution function from Chapter 2.
The logistic distribution has standard deviation π/√3 ≈ 1.81, the normal 1. So logit coefficients come out bigger, by a factor of roughly 1.6 to 1.8. The scale of y* is never observed, so only ratios of coefficients and probabilities mean anything.
A logit slope of 0.77 does not mean each extra hour raises the chance of passing by 77 percentage points. It raises the log-odds by 0.77. How much the probability moves depends on where you are on the S-curve: a lot in the middle, very little near 0 or 1. To talk in probabilities, compute partial effects, as below.
Maximum likelihood
OLS minimizes squared errors; that doesn’t fit an S-curve well, because a 0/1 outcome is not a curve plus normal noise. Instead, ask: for a given (β₀, β₁), how probable is the data we actually saw? Person i said yes with probability G(zi) and no with probability 1 − G(zi), where zi = β₀ + β₁xi. People are independent, so multiply, and take logs to turn the product into a sum:
Five people practiced 1, 2, 3, 4 and 5 hours; they failed, passed, failed, passed, passed (y = 0, 1, 0, 1, 1). Score the logit guess (β₀, β₁) = (−2, 1):
- Person 1: z = −1, Λ = 0.269. They failed, so add ln(1 − 0.269) = −0.313.
- Person 2: z = 0, Λ = 0.5, passed: ln 0.5 = −0.693. Person 3: z = 1, Λ = 0.731, failed: ln 0.269 = −1.313.
- Person 4: Λ(2) = 0.881, passed: −0.127. Person 5: Λ(3) = 0.953, passed: −0.049.
- Total: ℓ(−2, 1) = −2.495.
The guess (−1, 0.5) scores −2.656, and the “coin flip” (0, 0) scores 5 ln 0.5 = −3.466. The first guess explains the data better. The best possible is ℓ = −2.422 at (β̂₀, β̂₁) = (−2.65, 1.09), the maximum likelihood estimate.
Climbing to the top
There is no formula for the maximum, so computers climb. The gradient (the score) points uphill; the Hessian (the matrix of second derivatives) describes how the slope bends. Newton’s method fits a bowl to the hill at your current spot and jumps straight to the bowl’s top:
The logit log-likelihood is concave (it has one hill and no false peaks), so Newton’s method reliably finds the top in a handful of steps. Plain gradient ascent, which just takes a small step uphill each time, can crawl. Try both here.
The map uses the data from the simulation above, so edit the data there and the hill changes shape. Click anywhere on the map to start somewhere new.
Why maximum likelihood?
Under the model’s assumptions and in large samples, the MLE is consistent (it converges to the true β), asymptotically normal, and efficient: no other consistent estimator has a smaller variance in large samples. Its standard errors come from the curvature at the top: Var(β̂) ≈ [−∇²ℓ(β̂)]−1. A sharply peaked hill means a precise estimate; a flat ridge, an imprecise one. That’s why the 95% region on the map is so long and thin with raw x: the intercept and slope are hard to pin down separately, though the curve itself is well determined.
One case where it breaks: perfect separation. If some value of x splits all the yeses from all the nos, a steeper curve always fits a little better, and the likelihood keeps rising toward a cliff with no top. Software then reports enormous coefficients or fails to converge. The honest answer is that the data can’t tell you how steep the switch is.
Partial effects and odds ratios
For a continuous xj, the effect on the probability is the slope of the S-curve: ∂P(y = 1 | x)/∂xj = g(xβ) βj, where g is the density (Λ(1 − Λ) for logit, φ for probit). It changes with x, so economists report one of two summaries:
Plug in the sample averages: g(β̂₀ + x̄β̂₁) β̂₁. The effect for a made-up “average person”.
Average the effect over everyone: n−1 Σ g(β̂₀ + xiβ̂₁) β̂₁. Usually the better summary, and the one to compare with an LPM slope.
The odds are p/(1 − p). In a logit, raising x by 1 multiplies the odds by exp(β₁), wherever you start.
A logit for passing an exam gives β̂₀ = −3.69 and β̂₁ = 0.77 per hour (the default sample in the simulation).
- At 5 hours: z = −3.69 + 0.77 × 5 = 0.16, so P(pass) = Λ(0.16) = 0.540. The density is 0.540 × 0.460 = 0.248, and the partial effect is 0.248 × 0.77 = 0.191: one more hour adds about 19 percentage points.
- At 1 hour: z = −2.92, P = 0.051, density 0.049, effect 0.037: under 4 points per hour. Same coefficient, very different effect.
- Odds ratio: exp(0.77) = 2.16. At 5 hours the odds of passing are 0.540/0.460 = 1.17; at 6 hours P = Λ(0.93) = 0.717 and the odds are 2.53, which is 1.17 × 2.16.
Because many people in this sample practiced little or a lot, the average partial effect (0.090) is much smaller than the effect at the average (0.159). Reporting the APE answers “what happens on average if everyone practices one more hour?”
Goodness of fit and testing
Percent correctly predicted. Predict yes when p̂ ≥ 0.5 and count the hits. Simple, but it can mislead: if 95% of loans are repaid, predicting “repaid” for everyone scores 95% while learning nothing.
McFadden’s pseudo-R². Compare the log-likelihood with the one from a model with only an intercept: 1 − ℓur/ℓ0. It is 0 when x adds nothing; values of 0.2 to 0.4 already mean a good fit. It is not a share of variance explained.
Likelihood-ratio test. To test q restrictions (for example, that a coefficient is zero), fit the model with and without them and compare log-likelihoods:
In the default sample of 200 people, the logit’s log-likelihood is −74.48. With only an intercept (everyone gets the overall pass rate), it is −134.19.
- LR = 2 × (−74.48 − (−134.19)) = 119.4.
- With one restriction (β₁ = 0), the 5% critical value of χ²₁ is 3.84. 119.4 is far beyond it: p < 0.0001.
- Pseudo-R² = 1 − (−74.48)/(−134.19) = 0.445.
In the 5-person example above, LR = 2 × (−2.422 + 3.365) = 1.89 < 3.84: with five people, you can’t tell.
On the likelihood map, the white dashed line is the restricted model β₁ = 0 (when it fits on the map), and its best point is the intercept-only model. The LR statistic is twice the height difference between that point and the top.
A short history
Probits from poisons. Chester Bliss introduced the probit in 1934 to analyze how insects respond to doses of pesticide: the share killed rises along a normal S-curve in the log dose. (“Probit” is short for “probability unit”.)
Logits. Joseph Berkson proposed the logistic curve for the same kind of dose-response data in 1944 and coined the word “logit”. The same Berkson later described a famous selection bias, Berkson’s paradox, which comes back in Chapter 17. David Cox’s 1958 paper on the regression analysis of binary sequences helped make logistic regression a general-purpose tool.
Maximum likelihood. Ronald Fisher developed the method between 1912 and 1922 and showed why it works so well in large samples. Newton’s method for finding the top is much older, from Isaac Newton and Joseph Raphson in the late 1600s.
Choices as economics. Daniel McFadden built the logit into a theory of how people choose among options, the conditional logit (1974), and used it to forecast how many commuters would ride San Francisco’s BART trains before the system opened. He shared the 2000 Nobel memorial prize in economics with James Heckman; McFadden’s half was for his theory and methods for analyzing discrete choice.
Where it shows up
- Lending. Credit scorecards that estimate the chance a borrower defaults are traditionally built with logistic regression, partly because the odds-ratio form is easy to explain and audit.
- Marketing. “Will this customer respond to the offer?” and “will this subscriber cancel next month?” are binary outcomes; logit models predict them and their partial effects tell you what a discount is worth.
- Policy. Labor economists model whether people work (Wooldridge’s textbook example is married women’s labor force participation), and transport planners model which mode commuters choose.
- Sports. Win-probability graphs on sports broadcasts come from models of a binary outcome (win or lose) given the score, time left and field position.
- Medicine. Risk scores for heart disease and many other conditions are logistic regressions of “developed the condition” on age, blood pressure, cholesterol and so on.
Nothing here is financial or investment advice; the lending and marketing examples describe how models are built, not how to make decisions about money.
Practice
An LPM for buying an upgrade gives p̂ = 0.10 + 0.025 × discount (%). What does it predict for a 40% discount, and what’s wrong with that?
Show the answer
0.10 + 0.025 × 40 = 1.10, a 110% chance of buying, which is impossible. The LPM’s constant slope can’t hold all the way up; a logit or probit would bend toward 1 instead.
A logit gives P(buy) = Λ(−3 + 0.15 × discount). What is the probability of buying at a 20% discount?
Show the answer
z = −3 + 0.15 × 20 = 0, so P = Λ(0) = 0.5.
Using the same model, what is the partial effect of one more point of discount at 20%, and at 0%?
Show the answer
At 20%: Λ(0)(1 − Λ(0)) × 0.15 = 0.25 × 0.15 = 0.0375, or 3.75 percentage points. At 0%: z = −3, Λ = 0.047, density 0.047 × 0.953 = 0.045, effect 0.045 × 0.15 = 0.0068, under 1 point. The S-curve is steepest in the middle.
In the same logit, what is the odds ratio for 10 more points of discount?
Show the answer
exp(0.15 × 10) = exp(1.5) = 4.48. Ten more points of discount multiply the odds of buying by about 4.5, at any starting discount.
A probit gives P(approve) = Φ(−4 + 0.6 × score), with score in hundreds above 0. Find the partial effect of the score at score = 7.
Show the answer
z = −4 + 4.2 = 0.2; φ(0.2) = 0.391; effect = 0.391 × 0.6 = 0.235 per 100 points, or about 2.3 percentage points per 10 points of score near 700.
A model with three regressors has ℓ = −412.6. Dropping two of them gives ℓ = −418.9. Test whether the two matter, at 5%.
Show the answer
LR = 2 × (−412.6 + 418.9) = 12.6 with q = 2 restrictions. The 5% critical value of χ²₂ is 5.99, so reject: together the two regressors matter (p ≈ 0.002).
Your logit software reports a coefficient of 38.2 with a standard error of 4,105 after 25 iterations. What probably happened?
Show the answer
Perfect (or quasi-) separation: some value of that variable splits the outcomes perfectly, so the likelihood keeps rising as the coefficient grows, and the “estimate” is just where the algorithm stopped. Look at a cross-tabulation, consider whether the variable is effectively a proxy for the outcome, and use methods designed for separation (such as penalized likelihood) or say plainly in the report that the data are separated.
How the interactive works
The data. For each person, x is uniform over the middle 84% of the range, and y = 1 with probability Λ(β₁(x − m)) for logistic errors, or Φ(β₁(x − m)/1.81) for normal errors with the same spread, where m is the point where the chance is 50% and “how sharp the switch is” scales β₁.
The fits. The LPM uses the course library’s OLS with HC1 robust standard errors; logit and probit use its maximum likelihood (Newton’s method), with standard errors from the inverse Hessian. Before fitting, the page checks for separation, since then the maximum doesn’t exist. Partial effects and LR tests are computed from the formulas above.
The map. The log-likelihood is computed on a grid of about 170 × 120 points spanning 8 standard errors around the maximum each way; bands of color mark equal steps in log(1 + distance below the top). Newton steps halve their length if a full step would go downhill; gradient steps use a fixed rate of 1 over the largest eigenvalue of the information matrix at the top.
How it was checked (Node.js, before the page was built):
- Partial effects match numerical derivatives of the fitted probabilities to four decimals (logit APE 0.0896, PEA 0.1586; probit APE 0.0905, PEA 0.1508). The page’s Newton steps reach the library’s maximum to six decimals in 6 steps from (0, 0) and 8 steps from a start 1,843 log-likelihood units below the top.
- 1,000 samples of 100 people with no true effect: the LR test rejected 5.6% of the time at 5%. With a true effect, Wald 95% intervals for β₁ covered the truth 96.5% of the time; the logit/probit coefficient ratio averaged 1.74, and the LPM slope was on average 0.02 above the logit’s APE.
- Gradient ascent to within 0.01 of the top: 677 steps with raw x (condition number of the information matrix 208), 19 steps with centered x (condition number 7).
- Separation checks: {1, 2, 3, 4} with outcomes 0, 0, 1, 1 is flagged complete; {1, 2, 2, 4} with 0, 0, 1, 1 quasi; and the fit refuses to report logit coefficients for separated data.
Check yourself
Why can the linear probability model predict probabilities below 0 or above 1?
Show the answer
Any non-flat line eventually leaves [0, 1]. The S-shaped logit and probit curves flatten out instead.
In a logit with β̂₁ = 0.5, what does a one-unit increase in x do?
Show the answer
It adds 0.5 to the log-odds, so it multiplies the odds by exp(0.5). The change in probability depends on where you start; 0.5 × 0.25 is the change only at p = 0.5.
What does maximum likelihood choose?
Show the answer
The MLE maximizes the probability (likelihood) of the data actually observed, which for a binary outcome is the product of G(zi) for the yeses and 1 − G(zi) for the nos.
A logit has ℓ = −250 and the intercept-only model has ℓ = −260. What is the LR statistic for testing that all slopes are zero?
Show the answer
LR = 2(ℓur − ℓr) = 2(−250 + 260) = 20, compared with a χ² distribution whose degrees of freedom equal the number of slopes.
Why is the average partial effect usually preferred to the partial effect at the average?
Show the answer
The effect differs from person to person on an S-curve. The APE averages those effects; the PEA evaluates the curve at the mean of x, which may describe nobody (especially with dummy variables, where “0.4 of a college degree” isn’t a person).
Sources
- J. M. Wooldridge, Introductory Econometrics: A Modern Approach, 7th ed. (Cengage, 2020), section 7.5 (the linear probability model) and chapter 17.1 (logit and probit, partial effects, the likelihood ratio test, percent correctly predicted and pseudo-R²).
- J. H. Stock and M. W. Watson, Introduction to Econometrics, 4th ed. (Pearson, 2019), chapter 11 (regression with a binary dependent variable).
- Penn State Eberly College of Science, STAT 504: Analysis of Discrete Data (logistic regression, odds ratios and likelihood ratio tests).
- K. Train, Discrete Choice Methods with Simulation, 2nd ed. (Cambridge University Press, 2009), chapters 3 (logit) and 5 (probit); free online edition at UC Berkeley.
- C. I. Bliss, The method of probits, Science 79(2037), 38–39 (1934). J. Berkson, Application of the logistic function to bio-assay, Journal of the American Statistical Association 39(227), 357–365 (1944). D. R. Cox, The regression analysis of binary sequences, Journal of the Royal Statistical Society B 20(2), 215–242 (1958).
- R. A. Fisher, On the mathematical foundations of theoretical statistics, Philosophical Transactions of the Royal Society A 222, 309–368 (1922).
- D. McFadden, Conditional logit analysis of qualitative choice behavior, in P. Zarembka (ed.), Frontiers in Econometrics (Academic Press, 1974).
- Wikipedia (secondary): Logistic regression, Probit model, Maximum likelihood estimation, Daniel McFadden.