A derivative is a limit of difference quotients. A computer cannot take the limit, so it stops at a small $h$, and the Truncation Error and Order of Accuracy that leaves depends on which difference quotient we use. Taylor series tell us exactly how big it is.
- Forward: $f'(x) \approx \dfrac{f(x + h) - f(x)}{h}$.
- Backward: $f'(x) \approx \dfrac{f(x) - f(x - h)}{h}$.
- Central: $f'(x) \approx \dfrac{f(x + h) - f(x - h)}{2h}$.
- Second derivative: $f''(x) \approx \dfrac{f(x + h) - 2f(x) + f(x - h)}{h^{2}}$.
Key idea
The forward (and backward) difference has error $\frac{h}{2}f''(x) + O(h^{2})$: order $h$. The central difference has error $\frac{h^{2}}{6}f'''(x) + O(h^{4})$: order $h^{2}$. The second-difference formula has error $\frac{h^{2}}{12}f^{(4)}(x) + O(h^{4})$. For the same $h$ and about the same work, the central difference is usually far more accurate.
Why it works
Write the Taylor expansions $$f(x \pm h) = f(x) \pm hf'(x) + \frac{h^{2}}{2}f''(x) \pm \frac{h^{3}}{6}f'''(x) + \frac{h^{4}}{24}f^{(4)}(x) \pm \cdots$$ The forward quotient keeps the $\frac{h}{2}f''$ term. Subtracting the two expansions cancels every even power, so the central quotient's error starts at $\frac{h^{2}}{6}f'''$. Adding them cancels every odd power, which gives the second-difference formula with error $\frac{h^{2}}{12}f^{(4)}$. The symmetry of the central formulas is what buys the extra order.
Worked example: Forward and central side by side
For $f(x) = x^{3}$ at $x = 1$ (where $f'(1) = 3$), compute both differences with $h = 0.1$.
- Forward: $\frac{1.331 - 1}{0.1} = 3.31$, error $0.31$. The formula predicts $\frac{h}{2}f''(1) + \frac{h^{2}}{6}f'''(1) = 0.3 + 0.01$.
- Central: $\frac{1.331 - 0.729}{0.2} = 3.01$, error $0.01 = \frac{h^{2}}{6}f'''(1)$.
- Same three function values in play, thirty times smaller error.
Answer: $3.31$ and $3.01$
Worked example: Halving h
Repeat with $h = 0.05$.
- Forward: $\frac{1.157625 - 1}{0.05} = 3.1525$, error $0.1525$: about half of $0.31$ (order $h$).
- Central: $\frac{1.157625 - 0.857375}{0.1} = 3.0025$, error $0.0025$: a quarter of $0.01$ (order $h^{2}$).
Answer: errors $0.1525$ and $0.0025$
Worked example: A second derivative
Estimate $f''(1)$ for $f(x) = x^{4}$ with $h = 0.1$.
- $f(1.1) = 1.4641$, $f(1) = 1$, $f(0.9) = 0.6561$.
- $\frac{1.4641 - 2 + 0.6561}{0.01} = \frac{0.1202}{0.01} = 12.02$.
- Exact: $f''(1) = 12$. Error $0.02 = \frac{h^{2}}{12}f^{(4)}(1) = \frac{0.01 \times 24}{12}$.
Answer: $12.02$
Common mistake
Dividing the central difference by $h$ instead of $2h$. The two sample points $x - h$ and $x + h$ are $2h$ apart, so the slope between them divides by $2h$. Dividing by $h$ doubles the answer, a mistake no small $h$ will fix. Quick check: a forward difference with the same $h$ should come out close to your central value, not near half of it.
These formulas also turn differential equations into algebra: replacing $y''$ by the second difference is the starting point of finite-difference methods for boundary value problems. How small $h$ should be in floating point is the subject of The Best Step Size.