Econometrics · Chapter 2
Probability and Distributions
Every number in econometrics, from a poll to a regression slope, is a draw from some probability distribution. Here you get a laboratory of seven of them. Change their shapes, drop thousands of random draws into a live histogram, and drag a shaded band to measure probability. Then watch the most surprising fact in statistics happen in front of you: averages of almost anything, however lopsided, pile up into the same bell curve.
Distribution
Draw at random
Shade a probability
Live numbers
Missions
What you’re seeing
The green shape is a probability distribution: where a random quantity tends to land. For a continuous quantity it is a smooth curve called a density, and the probability of landing between two values is the area under the curve between them. For a count, it is a set of bars, and each bar’s height is the probability of that exact count.
The gold marks show the mean (the dashed line, the balance point of the shape) and one standard deviation either side (the bracket at the top), a measure of how spread out the shape is.
The draws fall in as gold dots, and the blue histogram counts where they landed, scaled so it can be compared with the curve. With a few dozen draws it is ragged; with thousands it hugs the curve. That is probability as long-run frequency.
The shaded band is a probability you choose. Drag the round handles a and b, or use the presets. The panel shows the exact probability from the formula and the share of your draws that actually landed in the band.
Random variables and their distributions
A random variable is a number whose value depends on chance: the roll of a die, the number of customers in the next hour, tomorrow’s change in the price of gasoline, the wage of a worker picked at random. Capital letters like X stand for the random variable; small letters like x for a value it can take.
Discrete random variables take separate values, usually counts. Their distribution is a list of probabilities p(x) = P(X = x) that add up to 1. The binomial counts successes in n independent tries; the Poisson counts rare events in a fixed stretch of time or space.
Continuous random variables can take any value in a range. Their distribution is a density f(x); probabilities are areas: P(a ≤ X ≤ b) = ∫ab f(x) dx. The probability of any one exact value is zero. Both kinds have a cumulative distribution function, F(x) = P(X ≤ x).
Expected value: the long-run average
For a fair die, E(X) = (1 + 2 + 3 + 4 + 5 + 6)/6 = 3.5, a value the die can never show. The expected value is not what you expect on one roll; it is the average over many. Three rules carry most of econometrics:
Variance: how far from the mean
Adding a constant shifts a distribution without spreading it, so b drops out. Doubling every value doubles the spread and quadruples the variance. For a fair die, E(X²) = (1 + 4 + 9 + 16 + 25 + 36)/6 = 91/6, so Var(X) = 91/6 − 3.5² = 35/12 ≈ 2.92 and σ ≈ 1.71.
An insurer sells a one-year policy for $300. With probability 1 in 100 the car is stolen and the insurer pays $20,000; otherwise it pays nothing. (These numbers are made up for the example.)
- Expected payout: 0.01 × $20,000 = $200, so the expected profit per policy is $100.
- Variance of the payout: 20,000² × 0.01 × 0.99 = 3,960,000 dollars², so σ ≈ $1,990. One policy is a wild gamble: usually +$300, sometimes −$19,700.
- Now sell 10,000 independent policies. The expected total payout is $2,000,000, and because variances of independent risks add, the standard deviation of the total is $1,990 × √10,000 = $199,000. To lose money, payouts must top the $3,000,000 in premiums, which is 5 standard deviations above the mean: by the normal approximation, a chance of less than one in a million.
The spread of the total grows like √n, while the expected profit grows like n. That gap is the law of large numbers at work, and it is the business model of insurance. (Real thefts aren’t perfectly independent, which is why insurers also worry about floods and storms that hit many customers at once.)
Covariance, correlation and conditional expectation
Covariance measures whether two random variables move together: Cov(X, Y) = E[(X − μX)(Y − μY)]. It is positive when above-average X tends to come with above-average Y. Its units are awkward (dollars times years, say), so we usually rescale it:
Independent variables have zero covariance. The reverse is not true: if X is −1, 0 or 1 with equal chances and Y = X², then Cov(X, Y) = E(X³) − E(X)E(X²) = 0, yet Y is completely determined by X. Correlation measures only straight-line association.
Conditional expectation is the average of Y among those with a given value of X: E(Y | X = x). The average wage of workers with 16 years of schooling is one number; with 12 years, another. As a function of x it is called the conditional expectation function, and it is the thing regression tries to estimate.
The overall average is the average of the group averages, weighted by group size. If 30% of workers have a degree and average $40 an hour, and the rest average $25, the overall average is 0.3 × 40 + 0.7 × 25 = $29.50.
The gold diamonds are E(wage | schooling) estimated from the data; the dashed curve is the true one used to simulate the workers. At each level there is a whole spread of wages, but the average traces a smooth, rising curve. Tick “Fit a straight line too”: the line is the best straight-line approximation to that curve, which is exactly what regression gives you (Chapter 4). The data are simulated, loosely patterned on U.S. wages.
The normal distribution and z-scores
The bell-shaped normal distribution is described by just its mean μ and standard deviation σ, written N(μ, σ²). Its density is f(x) = exp[−(x − μ)²/(2σ²)] ÷ (σ√(2π)). Any normal can be turned into the standard normal, N(0, 1), by measuring distance from the mean in standard deviations:
within 1 standard deviation of the mean (68.27%).
within 1.96 standard deviations (95.00%); within 2, 95.45%.
within 3 standard deviations (99.73%).
Two more facts make the normal central to econometrics. A linear function of a normal is normal, and a sum of independent normals is normal. And, by the central limit theorem below, sums and averages of many independent things are approximately normal even when the things themselves are not.
American men aged 20 and over average 68.9 inches tall (CDC, National Health and Nutrition Examination Survey, 2021–2023). Suppose heights are roughly normal with a standard deviation of about 3 inches (an assumption for this example). Six feet is 72 inches, so z = (72 − 68.9) ÷ 3 = 1.03. The share of a standard normal above 1.03 is about 15%.
Going the other way: the 90th percentile of a standard normal is 1.2816, so 90% of men would be shorter than 68.9 + 1.2816 × 3 ≈ 72.7 inches.
Counts, and the t and chi-square families
Binomial. With n independent tries, each a success with probability p, the number of successes has mean np and variance np(1 − p). Defective parts in a batch, customers who click an ad, free throws made.
Poisson. Counts of rare events in a fixed window: calls to a help line in a minute, typos on a page, claims on a policy. One number, λ, is both the mean and the variance, and P(X = k) = e−λ λk ÷ k!.
Chi-square with k degrees of freedom is the distribution of a sum of k squared independent standard normals, Z₁² + … + Zk². Each Z² has mean 1 and variance 2, so the sum has mean k and variance 2k. Sample variances and many test statistics follow it.
Student’s t with ν degrees of freedom is Z ÷ √(Q/ν), where Q is an independent chi-square with ν degrees of freedom. It is a bell with fatter tails, because dividing by an estimated spread adds uncertainty. As ν grows it becomes the standard normal.
Why they matter later. When you estimate a mean or a regression slope, you also have to estimate how noisy the data are. Dividing an estimate by its estimated standard error gives a t statistic, and with normal errors it follows exactly a t distribution (Chapters 3 and 5). Tests of several restrictions at once use the F distribution, a ratio of two chi-squares (Chapter 6). In large samples all of them settle into the normal and chi-square shapes.
In 1898 the statistician Ladislaus Bortkiewicz published counts of soldiers killed by horse kicks in the Prussian army corps over twenty years. In the ten corps usually analyzed there are 200 corps-years with 122 deaths, an average of λ = 0.61 per corps per year. If deaths are rare and independent, the Poisson formula predicts how many corps-years should show each count:
| Deaths in a year | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| Poisson prediction (200 × P) | 108.7 | 66.3 | 20.2 | 4.1 | 0.6 |
| Observed | 109 | 65 | 22 | 3 | 1 |
For example, 200 × e−0.61 = 200 × 0.543 = 108.7. The fit is remarkable for a formula with one number in it.
The central limit theorem
Take a sample of n independent draws from a population with mean μ and standard deviation σ, and compute their average X̄. Do it again and again. The averages have mean μ and standard deviation σ/√n, whatever the population looks like. And as n grows, their distribution approaches a normal curve:
Population
Sample size n
Goals
How fast is “large”? It depends on how lopsided the population is. The skewness of an average of n independent draws is the population’s skewness divided by √n. A flat population gives bell-shaped means with n = 5; waiting times (skewness 2) need around 30; incomes (skewness above 3) need more. The popular rule “n ≥ 30 is enough” is a rough guide, not a theorem.
Why it matters. The estimates in this course, means, proportions and regression slopes, are all averages of many small pieces. The central limit theorem is why their sampling distributions are close to normal in large samples, which is what makes confidence intervals and tests work even when the data themselves are not normal (Chapter 5).
A poll asks 1,000 people a yes-or-no question. Each answer is 1 (yes) or 0 (no); if the true share saying yes is 50%, each answer has σ = √(0.5 × 0.5) = 0.5. The share of yes answers in the poll is an average, so its standard deviation is 0.5/√1,000 = 0.0158, and by the central limit theorem it is close to normal. 95% of polls land within 1.96 × 0.0158 = 0.031 of the truth: the familiar “plus or minus 3 points.” To halve the margin, you need four times the people.
Waiting times at a help desk are exponential with mean 4 minutes (so σ = 4 too). What is the chance the average of 36 waits tops 5 minutes? The central limit theorem says X̄ ≈ N(4, (4/6)²), so P(X̄ > 5) ≈ P(Z > 1.5) = 0.067. The exact answer (a sum of exponentials has a gamma distribution) is 0.074. Close, but the skewed population still shows through at n = 36.
The law of large numbers
The central limit theorem describes the shape of the wobble in an average; the law of large numbers says the wobble dies out. As the number of independent draws grows, the sample average gets as close as you like to the population mean, with probability approaching 1. Jacob Bernoulli proved a version for coin-like events in Ars Conjectandi, published in 1713.
After ten heads in a row, a fair coin is not “due” for tails; the next toss is still 50–50. The law of large numbers works by swamping early streaks with many later tosses, not by correcting them. Watch the readout: the share of sixes closes in on 1/6, but the count of sixes minus the expected count does not shrink. It typically grows, like √n. Believing otherwise is called the gambler’s fallacy.
A short history, and the people
Games of chance. In 1654 Blaise Pascal and Pierre de Fermat worked out, in letters to each other, how to split the stakes of an unfinished game fairly, and with it the idea of expected value. Jacob Bernoulli’s Ars Conjectandi (1713) proved the first law of large numbers.
The bell. Abraham de Moivre found in 1733 that binomial probabilities for many tries follow a bell-shaped curve. Pierre-Simon Laplace generalized this into the central limit theorem, and Carl Friedrich Gauss used the normal curve for errors of measurement in 1809, which is why it is also called the Gaussian distribution.
Counts and small samples. Siméon Denis Poisson published his distribution in 1837. In 1908 William Sealy Gosset, a chemist at the Guinness brewery in Dublin, published the t distribution in Biometrika under the pen name “Student,” because the brewery’s experiments gave him only small samples and he needed exact answers from them. Karl Pearson introduced the chi-square test in 1900.
Foundations. In 1933 Andrey Kolmogorov set out the axioms that put probability on the same footing as the rest of mathematics. The same year, the first issue of Econometrica appeared.
Where you see it
Business. Insurance and warranties rest on the law of large numbers. Call centers and stores staff up using Poisson arrivals. Quality control counts defects with the binomial. Spotting a cluster of rare failures early, before it becomes a crisis, is a Poisson question: how many failures would chance alone produce? (See Firestone and the Ford Explorer.)
Markets. Daily stock returns have fatter tails than a normal curve: big moves happen far more often than a normal model predicts, which is one reason risk models built on the normal can fail in a crisis. Nothing here is financial or investment advice.
Polls and policy. Every margin of error in a news story is the central limit theorem. Government surveys, like the monthly jobs survey, publish standard errors computed the same way.
Sports and daily life. Goals in a soccer match are often modeled as Poisson counts. A batting average early in the season swings wildly and settles down by September: the law of large numbers.
Practice
A raffle. A ticket costs $5. One ticket in 1,000 wins $2,000; the rest win nothing. What is the expected value of buying a ticket?
Show the answer
Expected winnings: $2,000 × 1/1,000 = $2. Net of the price, the expected value is $2 − $5 = −$3 per ticket.
Rescaling. X has mean 10 and standard deviation 2. What are the mean, variance and standard deviation of Y = 3X − 5?
Show the answer
E(Y) = 3 × 10 − 5 = 25. Var(Y) = 3² × 2² = 36, so sd(Y) = 6. The −5 shifts but doesn’t spread.
Adding risks. X and Y have standard deviations 3 and 4. What is the standard deviation of X + Y if they are independent? If their correlation is 0.5? And of X − Y when independent?
Show the answer
Independent: Var = 9 + 16 = 25, sd = 5. With correlation 0.5, Cov = 0.5 × 3 × 4 = 6, so Var = 9 + 16 + 2 × 6 = 37, sd ≈ 6.08. X − Y when independent: Var = 9 + 16 = 25 again, sd = 5; subtracting an independent risk adds variance, it doesn’t cancel it.
z-scores. Test scores are normal with mean 500 and standard deviation 100. What share score above 650? What score marks the 90th percentile?
Show the answer
z = (650 − 500)/100 = 1.5, and P(Z > 1.5) = 0.067, about 6.7%. The 90th percentile of Z is 1.2816, so the score is 500 + 1.2816 × 100 ≈ 628.
A help line. Calls arrive at 4.5 per minute on average, as a Poisson count. What is the chance of a minute with no calls? With at most 2?
Show the answer
P(0) = e−4.5 = 0.011. P(≤ 2) = e−4.5(1 + 4.5 + 4.5²/2) = 0.0111 × 15.625 = 0.174.
Group averages. In a company, 40% of employees work in sales with an average salary of $70,000 and 60% in operations with an average of $55,000. What is the company’s average salary? Which rule did you use?
Show the answer
0.4 × 70,000 + 0.6 × 55,000 = 28,000 + 33,000 = $61,000. That is the law of iterated expectations: E(Y) = E[E(Y | X)].
The √n rule. A population has σ = 12. How large a sample do you need for the sample mean to have a standard deviation of 1? Of 0.5?
Show the answer
σ/√n = 1 needs √n = 12, so n = 144. Halving it to 0.5 needs √n = 24, n = 576: four times the sample for twice the precision.
Chi-square from scratch. If Z is standard normal, E(Z⁴) = 3. Use this to show that a chi-square with k degrees of freedom has mean k and variance 2k.
Show the answer
E(Z²) = Var(Z) + [E(Z)]² = 1. Var(Z²) = E(Z⁴) − [E(Z²)]² = 3 − 1 = 2. A chi-square is a sum of k independent Z², so its mean is k × 1 = k and, since variances of independent terms add, its variance is k × 2 = 2k. The mission “Mean k, variance 2k” checks this by simulation.
How the interactives work
The distributions. Densities, probabilities and cumulative probabilities come from exact formulas: the normal through the error function, the t and chi-square through the incomplete beta and gamma functions of the course’s shared statistics library, and binomial and Poisson probabilities through log-factorials. Normal draws use the polar method, exponential draws invert the cumulative distribution, Poisson draws invert it step by step, binomial draws count successes, chi-square draws add squared normals, and t draws divide a normal by the square root of an independent chi-square over its degrees of freedom. All random numbers come from a seeded generator, so a page reload starts the same way.
The central limit theorem draws n values from the chosen population and averages them. “Incomes” are lognormal (the log is normal with sd 0.75), “two humps” is an even mix of two normals at ±2 with sd 0.6, and “yes or no” is 1 with probability 0.1. The wages are simulated with log wage = 1.85 + 0.10 × schooling + noise (sd 0.45), so the true E(wage | schooling) = exp(1.85 + 0.10 × schooling + 0.45²/2).
How it was checked (node tests):
- Every density integrates to 1.0000 (every probability list sums to 1.00000), and the means and variances from numerical integration match the formulas to 4 decimals; the cumulative probabilities match the integrals to at least 5 × 10⁻⁷.
- 200,000 draws from each distribution: sample means and variances within 1% of the formulas (e.g. chi-square with 17 degrees of freedom: 17.02 and 34.09 vs 17 and 34).
- Normal areas: 0.68269, 0.95000 and 0.99730 within 1, 1.96 and 3 sd. Binomial(10, 0.5): P(4 ≤ X ≤ 6) = 0.65625, matching 672/1,024 exactly.
- Sample means, 20,000 per case: their spread matches σ/√n within 1%, and their skewness falls like skewness/√n (waiting times, n = 30: 0.38 vs 0.37).
- The largest gap between the t density with 30 degrees of freedom and the normal is 0.0045.
Check yourself
If E(X) = 5, what is E(2X + 3)?
Show the answer
E(2X + 3) = 2 E(X) + 3 = 13.
If Var(X) = 4, what is Var(2X + 3)?
Show the answer
Var(aX + b) = a² Var(X) = 4 × 4 = 16. The +3 shifts without spreading.
About 95% of a normal distribution lies within how many standard deviations of the mean?
Show the answer
±1.96 sd holds 95.0%; ±2 holds 95.45%, so “about 2” is the everyday version.
Two variables have a correlation of zero. Then…
Show the answer
Correlation only measures linear association. Y = X² with X symmetric around 0 has zero correlation with X.
The central limit theorem says that, for large samples…
Show the answer
It is about averages. The population keeps its shape; the means of many samples pile up into a bell with spread σ/√n.
A fair coin has landed heads ten times in a row. The chance of heads on the next toss is…
Show the answer
Tosses are independent. The law of large numbers dilutes streaks; it doesn’t correct them.
Sources
- OpenStax, Introductory Statistics 2e: Ch. 4 Discrete Random Variables, Ch. 6 The Normal Distribution, Ch. 7 The Central Limit Theorem.
- Penn State, STAT 414 Introduction to Probability Theory: Lesson 16, Normal Distributions and Lesson 27, The Central Limit Theorem.
- MIT OpenCourseWare, 18.05 Introduction to Probability and Statistics (Spring 2022).
- J. M. Wooldridge, Introductory Econometrics: A Modern Approach, 7th ed. (Cengage, 2020), Math Refresher B, Fundamentals of Probability (Appendix B in earlier editions). J. H. Stock and M. W. Watson, Introduction to Econometrics, 4th ed., Chapter 2.
- CDC National Center for Health Statistics, FastStats: Body Measurements (mean height of men 20 and over, 2021–2023).
- “Student” (W. S. Gosset), “The Probable Error of a Mean,” Biometrika 6 (1908). L. von Bortkiewicz, Das Gesetz der kleinen Zahlen (Leipzig, 1898); the horse-kick table is transcribed by the University of York’s history of statistics pages, Bortkiewicz on horse-kicks (the ten corps in the example leave out corps G, I, VI and XI).