Study
Deep Learning · Weight initialisation
Why initial weight scale matters, Xavier/Glorot for tanh and He/Kaiming for ReLU, and watching activations explode or collapse through 50 layers.
- Initial weights should keep the variance of activations roughly constant from layer to layer.
- He initialisation scales by √(2/fan_in) to compensate for ReLU zeroing half its inputs.