Deep Learning · Dropout

Dropout (Srivastava et al., 2014) zeroes each activation with probability p during training. The network can’t rely on any single unit, which works a bit like training an ensemble of thinned networks.

• PyTorch uses inverted dropout: surviving activations are multiplied by 1/(1−p), so the expected value is unchanged and nothing needs rescaling at test time.
• In eval() mode, dropout does nothing.

To test it, I overfit on purpose: a wide MLP on only 200 noisy training points, trained long enough to memorise them. Then I compared test accuracy with p = 0 and p = 0.5.

import torch, torch.nn as nn
from sklearn.datasets import make_moons

drop = nn.Dropout(p=0.5)
x = torch.ones(10)
drop.train(); print("train mode:", drop(x).tolist())
drop.eval();  print("eval mode :", drop(x).tolist())

Xtr, ytr = make_moons(200, noise=0.35, random_state=0)
Xte, yte = make_moons(2000, noise=0.35, random_state=1)
Xtr, ytr, Xte, yte = map(torch.tensor, (Xtr, ytr, Xte, yte))
Xtr, Xte = Xtr.float(), Xte.float()

for p in (0.0, 0.5):
    torch.manual_seed(0)
    net = nn.Sequential(nn.Linear(2, 512), nn.ReLU(), nn.Dropout(p), nn.Linear(512, 512), nn.ReLU(), nn.Dropout(p), nn.Linear(512, 2))
    opt = torch.optim.Adam(net.parameters(), lr=1e-3)
    for _ in range(2000):
        opt.zero_grad()
        nn.functional.cross_entropy(net(Xtr), ytr).backward()
        opt.step()
    net.eval()
    with torch.no_grad():
        tr = (net(Xtr).argmax(1) == ytr).float().mean().item()
        te = (net(Xte).argmax(1) == yte).float().mean().item()
    print(f"dropout p={p}: train acc {tr:.3f} | test acc {te:.3f}")

Output:

train mode: [2.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 2.0, 0.0, 0.0]
eval mode : [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
dropout p=0.0: train acc 1.000 | test acc 0.828
dropout p=0.5: train acc 0.950 | test acc 0.846