To solve several equations in several unknowns, $\mathbf{F}(\mathbf{x}) = \mathbf{0}$, Newton's idea carries over: replace $\mathbf{F}$ by its linear approximation and solve that. The derivative becomes a matrix of Partial Derivatives, and dividing by it becomes solving a linear system, which you know from Inverses by Gauss–Jordan. Its convergence matches Newton's method in one variable.
For $\mathbf{F} = (F_1, \ldots, F_n)$, the Jacobian $J(\mathbf{x})$ is the $n \times n$ matrix with $$J_{ij} = \frac{\partial F_i}{\partial x_j}:$$ row $i$ holds the partial derivatives of $F_i$.
Key idea
Why it works
Worked example: One step
- $\mathbf{F} = (x^{2} + y^{2} - 4,\ xy - 1)$ and $J = \begin{bmatrix} 2x & 2y \\ y & x \end{bmatrix}$.
- At $(2, 1)$: $\mathbf{F} = (1, 1)$ and $J = \begin{bmatrix} 4 & 2 \\ 1 & 2 \end{bmatrix}$.
- Solve $\begin{bmatrix} 4 & 2 \\ 1 & 2 \end{bmatrix}\boldsymbol{\Delta} = \begin{bmatrix} -1 \\ -1 \end{bmatrix}$: subtracting the equations gives $3\Delta_1 = 0$, so $\Delta_1 = 0$ and $\Delta_2 = -\frac{1}{2}$.
- $\mathbf{x}_1 = (2, 0.5)$.
Answer: $(2, 0.5)$
Worked example: Watching it converge
- At $(2, 0.5)$: $\mathbf{F} = (0.25, 0)$ and $J = \begin{bmatrix} 4 & 1 \\ 0.5 & 2 \end{bmatrix}$, giving $\boldsymbol{\Delta} \approx (-0.0667, 0.0167)$ and $\mathbf{x}_2 \approx (1.93333, 0.51667)$.
- One more step: $\mathbf{x}_3 \approx (1.931853, 0.517637)$.
- The root is $x = \sqrt{2 + \sqrt{3}} \approx 1.931852$, $y = \frac{1}{x} \approx 0.517638$. Errors: about $7 \times 10^{-2}$, $1.5 \times 10^{-3}$, $1 \times 10^{-6}$: digits roughly doubling.
Answer: $\approx (1.931852, 0.517638)$
Each step costs more than in one variable. Building $J$ takes $n^{2}$ partial derivatives, and solving the linear system by elimination takes about $\frac{2}{3}n^{3}$ operations (see The Cost of Elimination). For large systems, methods that update an approximate Jacobian instead of recomputing it (quasi-Newton methods) are common.
Common mistake