Deep Learning · Multilayer perceptron

A multilayer perceptron (MLP) stacks layers:

$\mathbf{h} = \phi(W_1\mathbf{x} + \mathbf{b}_1), \qquad \hat{y} = W_2\mathbf{h} + b_2$

The activation φ (ReLU, tanh, sigmoid) is essential. Without it, $W_2(W_1\mathbf{x})$ is just another linear map, and stacking layers buys nothing.

To show the hidden layer solves XOR, I set the weights by hand: one hidden unit acts as OR, another as AND, and the output computes “OR and not AND”. Training them automatically comes next, with backpropagation.

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
step = lambda z: (z > 0).astype(int)

W1 = np.array([[1, 1],     # hidden unit 1 ~ OR
               [1, 1]])    # hidden unit 2 ~ AND
b1 = np.array([-0.5, -1.5])
W2 = np.array([1, -2])     # OR minus 2*AND
b2 = -0.5

H = step(X @ W1.T + b1)
out = step(H @ W2 + b2)
print("hidden activations:\n", H)
print("XOR output:", out)

# without the non-linearity, two layers are one linear map
A, B = np.random.default_rng(0).normal(size=(3, 2)), np.random.default_rng(1).normal(size=(1, 3))
x = np.array([0.3, -1.2])
print("B(Ax) == (BA)x:", np.allclose(B @ (A @ x), (B @ A) @ x))

Output:

hidden activations:
 [[0 0]
 [1 0]
 [1 0]
 [1 1]]
XOR output: [0 1 1 0]
B(Ax) == (BA)x: True