Deep Learning · Backpropagation

Training needs the gradient of the loss with respect to every weight. Backpropagation (popularised by Rumelhart, Hinton and Williams in 1986) gets them all in one backward pass, using the chain rule:

1. Forward: compute each layer’s output and keep the intermediate values.
2. Backward: start from ∂L/∂ŷ. At each layer, multiply by the local derivative, which gives the gradients for that layer’s weights and passes the signal back to the previous layer.

I wrote it by hand for a 2-4-1 network with a tanh hidden layer and a sigmoid output, trained on XOR. Then I checked one gradient numerically with $(L(w+\epsilon)-L(w-\epsilon))/2\epsilon$.

import numpy as np

rng = np.random.default_rng(1)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)
W1, b1 = rng.normal(0, 1, (2, 4)), np.zeros(4)
W2, b2 = rng.normal(0, 1, (4, 1)), np.zeros(1)
sig = lambda z: 1 / (1 + np.exp(-z))

def forward(W1, b1, W2, b2):
    h = np.tanh(X @ W1 + b1)
    p = sig(h @ W2 + b2)
    loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
    return h, p, loss

for step in range(3001):
    h, p, loss = forward(W1, b1, W2, b2)
    dz2 = (p - y) / len(X)                 # dL/dz for sigmoid + cross-entropy
    dW2, db2 = h.T @ dz2, dz2.sum(0)
    dh = dz2 @ W2.T
    dz1 = dh * (1 - h ** 2)                # tanh'
    dW1, db1 = X.T @ dz1, dz1.sum(0)
    if step == 0:                          # gradient check on one weight
        eps = 1e-5; Wp, Wm = W1.copy(), W1.copy(); Wp[0, 0] += eps; Wm[0, 0] -= eps
        num = (forward(Wp, b1, W2, b2)[2] - forward(Wm, b1, W2, b2)[2]) / (2 * eps)
        print(f"grad check: backprop {dW1[0, 0]:.6f} vs numerical {num:.6f}")
    for P, G in [(W1, dW1), (b1, db1), (W2, dW2), (b2, db2)]:
        P -= 1.0 * G
    if step % 1000 == 0:
        print(f"step {step}: loss {loss:.4f}")
print("predictions:", forward(W1, b1, W2, b2)[1].ravel().round(3))

Output:

grad check: backprop -0.029283 vs numerical -0.029283
step 0: loss 0.6712
step 1000: loss 0.0024
step 2000: loss 0.0012
step 3000: loss 0.0008
predictions: [0.    0.999 0.999 0.001]

One honest note: with my first random seed (0), this exact network got stuck. Loss plateaued around 0.35 and two of the four outputs sat at 0.5. Seeds 1 to 5 all solved XOR. Even tiny networks can land in poor local minima, which is one reason initialisation and optimisers get their own posts.