A neural network sounds exotic, but you already know every part of it: matrix-vector products, ramps and composition. Networks built this way, with many more units and with weights found by training, are the math inside modern speech and image recognition and language models. Here is the whole idea at a size you can trace by hand.
Key idea
A layer takes a vector $\mathbf{x}$, computes $\mathbf{z} = W\mathbf{x} + \mathbf{b}$, then applies ReLU to each entry: $\mathbf{h} = \operatorname{ReLU}(W\mathbf{x} + \mathbf{b})$. Each hidden unit is a switch: on when its $z$ is positive (it passes $z$ through), off when $z \le 0$ (it outputs $0$). A two-layer network with one output computes $$y = \mathbf{v} \cdot \operatorname{ReLU}(W\mathbf{x} + \mathbf{b}) + c.$$
Why it works
Why the switches matter: without them, $$\mathbf{v} \cdot (W\mathbf{x} + \mathbf{b}) + c = (\mathbf{v}^{T}W)\,\mathbf{x} + (\mathbf{v} \cdot \mathbf{b} + c),$$ a single affine map, which is just a line or a plane. A composition of affine maps is affine, so stacking any number of switch-free layers never makes anything new. The ReLU between the layers breaks this. With one input, hidden unit $j$ adds the ramp $v_{j}\operatorname{ReLU}(w_{j}x + b_{j})$, which bends at $x = -b_{j}/w_{j}$, so the output is a sum of ramps: a bent-line graph with at most one bend per unit.
Worked examples
Worked example: A forward pass by hand
Let $W = \begin{bmatrix} 1 & 2 \\ -1 & 1 \end{bmatrix}$, $\mathbf{b} = \begin{bmatrix} 0 \\ 1 \end{bmatrix}$, $\mathbf{v} = (2, 5)$ and $c = -1$. Find $y$ for $\mathbf{x} = \begin{bmatrix} 3 \\ -1 \end{bmatrix}$.
- $$W\mathbf{x} = \begin{bmatrix} 1 \cdot 3 + 2 \cdot (-1) \\ (-1) \cdot 3 + 1 \cdot (-1) \end{bmatrix} = \begin{bmatrix} 1 \\ -4 \end{bmatrix}.$$
- Add the bias: $\mathbf{z} = \begin{bmatrix} 1 \\ -3 \end{bmatrix}$.
- Switches: $\mathbf{h} = \operatorname{ReLU}(\mathbf{z}) = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$. Unit 2 is off.
- Output: $y = 2 \cdot 1 + 5 \cdot 0 - 1 = 1$. For this input the weight $5$ never mattered.
Answer: $y = 1$
Worked example: Turning a rule into weights
Build a network, with no training, that gives $10$ when $x \ge 5$ and $12$ otherwise, for every whole number $x$.
- A sum of ramps is continuous, so it cannot jump. But it can slide down by $2$ between $x = 4$ and $x = 5$, where there are no whole numbers to get wrong.
- $\operatorname{ReLU}(x - 4) - \operatorname{ReLU}(x - 5)$ is $0$ for $x \le 4$, climbs to $1$ at $x = 5$, and then stays at $1$, because after $x = 5$ both ramps rise together.
- So $y = 12 - 2\operatorname{ReLU}(x - 4) + 2\operatorname{ReLU}(x - 5)$. As a network: $W = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$, $\mathbf{b} = \begin{bmatrix} -4 \\ -5 \end{bmatrix}$, $\mathbf{v} = (-2, 2)$ and $c = 12$.
- Check: $x = 3$ gives $12$; $x = 5$ gives $12 - 2 \cdot 1 + 2 \cdot 0 = 10$; $x = 9$ gives $12 - 2 \cdot 5 + 2 \cdot 4 = 10$.
- Between $4$ and $5$ the output is a ramp: $x = 4.5$ gives $11$, a value the rule never gives. The network is exact on the inputs it was built for, the whole numbers, and only there.
Answer: $y = 12 - 2\operatorname{ReLU}(x - 4) + 2\operatorname{ReLU}(x - 5)$
This idea scales up. Handwired, a project by Cheng Lou, compiles ordinary if-else and loop code into fixed network weights built from affine maps and ReLU switches like these, with no training at all. The weights are exact for the inputs the program supports, and because they are ordinary weights, training could still adjust them later.
Worked example: Logic with one switch
The inputs $x_{1}$ and $x_{2}$ are bits, $0$ or $1$. What does $y = x_{1} + x_{2} - 2\operatorname{ReLU}(x_{1} + x_{2} - 1)$ compute?
- The ramp $\operatorname{ReLU}(x_{1} + x_{2} - 1)$ is $1$ when both bits are $1$, and $0$ otherwise.
- Table: $(0, 0) \to 0$, $(0, 1) \to 1$, $(1, 0) \to 1$, $(1, 1) \to 2 - 2 = 0$.
- That is XOR: $1$ when exactly one bit is $1$. No affine map $px_{1} + qx_{2} + r$ can do this, because its values always satisfy $y(0, 0) + y(1, 1) = y(0, 1) + y(1, 0)$, while XOR has $0 + 0 \ne 1 + 1$. The switch is what makes it possible.
Answer: XOR (exclusive or)
Common mistake
Switching before adding the bias. $\operatorname{ReLU}(W\mathbf{x}) + \mathbf{b}$ is not a layer. The bias goes in first, because it decides where each switch flips. In the first example, $\operatorname{ReLU}(W\mathbf{x}) + \mathbf{b} = (1, 1)$, but the layer gives $(1, 0)$. Check: every hidden output should be $0$ or positive.
Common mistake
Expecting more layers to help without switches. Two layers with no ReLU collapse into one affine map, as the box above shows. Check: if your network has no switches, multiply it out; you will get one matrix and one bias.
The output layer usually has no switch, so $y$ can be negative. Training means choosing $W$, $\mathbf{b}$, $\mathbf{v}$ and $c$ to fit examples. It uses gradient descent, with the derivatives found by backpropagation. The picture below does it on a small data set.