A moment generating function packs all of a distribution's moments, $E[X], E[X^{2}], \ldots$, into one function. It turns the hard convolution of independent variables into a simple product, and it is the main tool for proving the central limit theorem. The key fact is the exponential series from known series, and the moments are the expected values of functions you already compute.
Key idea
The moment generating function of $X$ is $M_X(t) = E\left[e^{tX}\right]$. Its derivatives at 0 are the moments: $M'(0) = E[X]$, $M''(0) = E[X^{2}]$, and so on, so $\operatorname{Var}(X) = M''(0) - (M'(0))^{2}$. For independent $X$ and $Y$, $M_{X + Y}(t) = M_X(t)M_Y(t)$. Two variables with the same MGF (near $t = 0$) have the same distribution.
Why it works
Expand: $e^{tX} = 1 + tX + \frac{t^{2}X^{2}}{2!} + \cdots$, so by linearity $$M(t) = 1 + tE[X] + \frac{t^{2}}{2!}E[X^{2}] + \frac{t^{3}}{3!}E[X^{3}] + \cdots,$$ and the $k$th derivative at $t = 0$ picks out $E[X^{k}]$. For independent $X$ and $Y$, $e^{t(X + Y)} = e^{tX}e^{tY}$ is a product of independent factors, whose expected value is the product of the expected values.
| Distribution | $M(t)$ | Mean | Variance |
|---|---|---|---|
| Bernoulli($p$) | $1 - p + pe^{t}$ | $p$ | $p(1 - p)$ |
| Binomial($n, p$) | $(1 - p + pe^{t})^{n}$ | $np$ | $np(1 - p)$ |
| Poisson($\lambda$) | $e^{\lambda(e^{t} - 1)}$ | $\lambda$ | $\lambda$ |
| Exponential (rate $\lambda$) | $\frac{\lambda}{\lambda - t}$, $t < \lambda$ | $\frac{1}{\lambda}$ | $\frac{1}{\lambda^{2}}$ |
| Normal $N(\mu, \sigma^{2})$ | $e^{\mu t + \sigma^{2}t^{2}/2}$ | $\mu$ | $\sigma^{2}$ |
Worked examples
Worked example: A discrete MGF
$M(t) = \frac{1}{4}e^{-t} + \frac{1}{2} + \frac{1}{4}e^{2t}$. Find $E[X]$ and $\operatorname{Var}(X)$.
- This $X$ is $-1$, 0 or 2 with probabilities $\frac{1}{4}, \frac{1}{2}, \frac{1}{4}$.
- $M'(t) = -\frac{1}{4}e^{-t} + \frac{1}{2}e^{2t}$, so $E[X] = -\frac{1}{4} + \frac{1}{2} = \frac{1}{4}$.
- $M''(t) = \frac{1}{4}e^{-t} + e^{2t}$, so $E[X^{2}] = \frac{5}{4}$ and $\operatorname{Var}(X) = \frac{5}{4} - \frac{1}{16} = \frac{19}{16}$.
Answer: $\frac{1}{4}$ and $\frac{19}{16}$
Worked example: The Poisson
Find the mean and variance from $M(t) = e^{\lambda(e^{t} - 1)}$.
- $M'(t) = \lambda e^{t}M(t)$, so $M'(0) = \lambda$.
- $M''(t) = \lambda e^{t}M(t) + \lambda e^{t}M'(t)$, so $M''(0) = \lambda + \lambda^{2}$.
- $\operatorname{Var}(X) = \lambda + \lambda^{2} - \lambda^{2} = \lambda$.
Answer: Both are $\lambda$.
Worked example: Recognizing a sum
$X$ and $Y$ are independent Poisson with means 2 and 3. Identify the distribution of $X + Y$.
- $$M_{X + Y}(t) = e^{2(e^{t} - 1)}e^{3(e^{t} - 1)} = e^{5(e^{t} - 1)}$$
- That is the Poisson MGF with $\lambda = 5$, and an MGF determines the distribution.
Answer: Poisson with mean 5, matching the convolution lesson with far less work.
Common mistake
Reading $M''(0)$ as the variance. It is $E[X^{2}]$, the second moment about 0; the variance needs $(M'(0))^{2}$ subtracted. Check with a constant $X = 3$: $M(t) = e^{3t}$, $M''(0) = 9$, but the variance is 0.