Quadratic Forms and Positive Definite Matrices

Expressions like $5x_1^{2} + 4x_1x_2 + 2x_2^{2}$ appear whenever we measure energy, variance or curvature. Every such expression is $\mathbf{x}^{T}A\mathbf{x}$ for a symmetric matrix $A$, and the spectral theorem tells us exactly what it looks like.

A quadratic form on $\mathbb{R}^n$ is $Q(\mathbf{x}) = \mathbf{x}^{T}A\mathbf{x}$ with $A$ symmetric. The diagonal entries of $A$ are the coefficients of the squares, and each cross term's coefficient is split evenly between two off-diagonal entries: $$5x_1^{2} + 4x_1x_2 + 2x_2^{2} = \mathbf{x}^{T}\begin{bmatrix} 5 & 2 \\ 2 & 2 \end{bmatrix}\mathbf{x}.$$

Key idea

Principal axes. With $A = PDP^{T}$ and new coordinates $\mathbf{y} = P^{T}\mathbf{x}$, $Q = \lambda_1y_1^{2} + \cdots + \lambda_ny_n^{2}$: no cross terms. So $Q$ is positive definite ($Q(\mathbf{x}) > 0$ for all $\mathbf{x} \neq \mathbf{0}$) exactly when every eigenvalue of $A$ is positive, negative definite when all are negative, and indefinite when there are eigenvalues of both signs.

Why it works

$\mathbf{x}^{T}A\mathbf{x}$ $= \mathbf{x}^{T}PDP^{T}\mathbf{x}$ $= (P^{T}\mathbf{x})^{T}D(P^{T}\mathbf{x})$ $= \mathbf{y}^{T}D\mathbf{y}$ $= \sum \lambda_iy_i^{2}$. Since $P$ is invertible, $\mathbf{x} \neq \mathbf{0}$ exactly when $\mathbf{y} \neq \mathbf{0}$. If every $\lambda_i > 0$, the sum is positive for every nonzero $\mathbf{y}$. If some $\lambda_i \leq 0$, choose $\mathbf{y} = \mathbf{e}_i$ to get $Q = \lambda_i \leq 0$. Geometrically, the eigenvectors are the axes of the level curves $Q = c$ (ellipses or hyperbolas).

Worked example: Classify a form

Classify $Q = 5x_1^{2} + 4x_1x_2 + 2x_2^{2}$ and find its principal-axes form.

$A = \begin{bmatrix} 5 & 2 \\ 2 & 2 \end{bmatrix}$: trace $7$, determinant $6$, so $\lambda^{2} - 7\lambda + 6 = (\lambda - 6)(\lambda - 1)$.
Both eigenvalues positive: positive definite. Every level curve $Q = c > 0$ is an ellipse.
Unit eigenvectors: $\frac{1}{\sqrt{5}}(2, 1)$ for $6$ and $\frac{1}{\sqrt{5}}(-1, 2)$ for $1$. In those coordinates $Q = 6y_1^{2} + y_2^{2}$.

Answer: Positive definite; $Q = 6y_1^2 + y_2^2$.

Worked example: An indefinite form

Classify $Q = x^{2} + 6xy + y^{2}$.

$A = \begin{bmatrix} 1 & 3 \\ 3 & 1 \end{bmatrix}$ has eigenvalues $4$ (along $(1, 1)$) and $-2$ (along $(1, -1)$).
One positive, one negative: indefinite. For example $Q(1, 1) = 8 > 0$ and $Q(1, -1) = -4 < 0$. The graph is a saddle.

Answer: Indefinite.

The graph of $Q = ax^2 + 2bxy + cy^2$, the form of $\begin{bmatrix} a & b \\ b & c \end{bmatrix}$. Positive definite gives a bowl; change $b$ until $ac - b^2 < 0$ and it becomes a saddle.

For a $2 \times 2$ matrix $\begin{bmatrix} a & b \\ b & c \end{bmatrix}$ there is a shortcut: the product of the eigenvalues is $\det A = ac - b^{2}$ and their sum is $a + c$. So $Q$ is positive definite exactly when $a > 0$ and $ac - b^{2} > 0$. This is the test behind the second derivative test in multivariable calculus, where $A$ is the matrix of second partial derivatives.

A positive semidefinite form ($Q \geq 0$, with some eigenvalue $0$) is flat along the eigenvectors for $0$: its graph is a trough rather than a bowl. Every matrix of the form $A^{T}A$ is positive semidefinite, since $\mathbf{x}^{T}A^{T}A\mathbf{x} = \lVert A\mathbf{x}\rVert^{2} \geq 0$, and positive definite exactly when the columns of $A$ are independent. That fact is used in the next topic.

Common mistake

Putting the whole cross-term coefficient in both off-diagonal spots. For $4x_1x_2$ the entries are $2$ and $2$, because $\mathbf{x}^{T}A\mathbf{x}$ picks up $a_{12}x_1x_2 + a_{21}x_2x_1$ $= 2a_{12}x_1x_2$. Check: multiply out $\mathbf{x}^{T}A\mathbf{x}$ with your matrix and compare.

Quadratic Forms and Positive Definite Matrices

Practice

Builds on

Leads to

Sources and further reading