Study
Deep Learning · Activation functions
Sigmoid, tanh, ReLU, Leaky ReLU and GELU compared by shape and gradient, and why saturating activations make deep networks hard to train.
- Sigmoid and tanh saturate, so their gradients vanish for large inputs.
- ReLU's gradient is exactly 1 for positive inputs, which keeps signals flowing in deep nets.