Do $X$ and $Y$ tend to be large together, or does one tend to be large when the other is small? Covariance measures that tendency, using the variance shortcut idea. It is computed from a joint distribution, and scaled, it becomes the correlation $\rho$.
Key idea
$$\operatorname{Cov}(X, Y) = E[(X - \mu_X)(Y - \mu_Y)] = E[XY] - E[X]E[Y], \qquad \rho = \frac{\operatorname{Cov}(X, Y)}{\sigma_X\sigma_Y},$$ with $-1 \le \rho \le 1$. Independent variables have covariance 0, but covariance 0 does not imply independence.
Why it works
$(X - \mu_X)(Y - \mu_Y)$ is positive when both are above (or both below) their means, and negative when they are on opposite sides; its average measures the trend. Expanding and using linearity gives the shortcut. For $\lvert \rho \rvert \le 1$: let $U = \frac{X}{\sigma_X}$ and $V = \frac{Y}{\sigma_Y}$. Then $0 \le \operatorname{Var}(U + V) = 1 + 1 + 2\rho$ and $0 \le \operatorname{Var}(U - V) = 2 - 2\rho$, so $-1 \le \rho \le 1$. If $X$ and $Y$ are independent, $$E[XY] = \sum xy\,p_X(x)p_Y(y) = E[X]E[Y]$$ so the covariance is 0. The converse fails because covariance only sees straight-line trends.
Worked examples
Worked example: From a table
$P(X = 0, Y = 0) = 0.4$, $P(X = 0, Y = 1) = 0.1$, $P(X = 1, Y = 0) = 0.2$, $P(X = 1, Y = 1) = 0.3$. Find $\operatorname{Cov}(X, Y)$ and $\rho$.
- $E[X] = 0.5$, $E[Y] = 0.4$, and $E[XY] = P(X = 1, Y = 1) = 0.3$.
- $\operatorname{Cov}(X, Y) = 0.3 - 0.5 \times 0.4 = 0.1$.
- $\operatorname{Var}(X) = 0.25$, $\operatorname{Var}(Y) = 0.24$, so $\rho = \frac{0.1}{\sqrt{0.25 \times 0.24}} \approx 0.41$.
Answer: $0.1$ and about $0.41$
Worked example: Correlation from the pieces
$\operatorname{Var}(X) = 4$, $\operatorname{Var}(Y) = 9$, $\operatorname{Cov}(X, Y) = -3$. Find $\rho$, and $\operatorname{Cov}(2X + 1, 5 - Y)$.
- $\rho = \frac{-3}{2 \cdot 3} = -\frac{1}{2}$.
- Constants drop out and factors multiply: $$\operatorname{Cov}(2X + 1, 5 - Y) = 2 \cdot (-1) \cdot (-3) = 6$$
Answer: $-\frac{1}{2}$ and 6
Worked example: Zero covariance, dependent
$X$ is $-1$, 0 or 1 with probability $\frac{1}{3}$ each, and $Y = X^{2}$. Find $\operatorname{Cov}(X, Y)$.
- $E[X] = 0$, and $$E[XY] = E[X^{3}] = \frac{-1 + 0 + 1}{3} = 0$$
- $\operatorname{Cov}(X, Y) = 0 - 0 \cdot E[Y] = 0$.
- Yet $$P(X = 0, Y = 1) = 0 \ne P(X = 0)P(Y = 1) = \frac{1}{3} \cdot \frac{2}{3}$$
Answer: 0, although $Y$ depends on $X$ completely.
Common mistake
Concluding independence from zero covariance. Covariance measures only a linear trend; a perfect curved relationship, as above, can have covariance 0. Check: independence needs every cell (or the density everywhere) to factor; covariance 0 is only a consequence of it.