Study
Deep Learning · Optimisers: SGD, momentum, Adam
Mini-batch SGD, momentum, RMSProp-style scaling and Adam compared on an ill-conditioned valley, using torch.optim.
- Momentum smooths noisy gradients and speeds up travel along long, shallow valleys.
- Adam adapts a separate step size per parameter from running averages of the gradient and its square.