Learning in public, one day at a time

Latest

• 22 Apr 2025 · Study

  Deep Learning · LSTM and GRU

  Gated recurrent units: the LSTM's cell state and three gates, the simpler GRU, and an experiment measuring how far gradients reach back through 100 time steps.

• 11 Apr 2025 · Study

  Deep Learning · RNN

  Recurrent networks carry a hidden state across time steps; the unrolled view, nn.RNN shapes, and why gradients vanish over long sequences.

• 3 Apr 2025 · Study

  Deep Learning · CNN intro

  Convolutions as sliding filters with shared weights, padding, stride, pooling, output-size arithmetic, and a small CNN versus an MLP on digits.

• 25 Mar 2025 · Study

  Deep Learning · PyTorch training loop

  The standard PyTorch loop with Dataset, DataLoader, nn.Module, a loss, an optimiser and an evaluation pass, applied to scikit-learn's handwritten digits.

• 14 Mar 2025 · Study

  Deep Learning · PyTorch tensors and autograd

  PyTorch tensors (NumPy-like, with devices and dtypes), the dynamic computation graph, requires_grad, backward(), and no_grad.

• 6 Mar 2025 · Study

  Deep Learning · Dropout

  Dropout randomly zeroes activations during training as a regulariser; inverted scaling, train vs eval behaviour, and its effect on an overfitting MLP.

• 25 Feb 2025 · Study

  Deep Learning · Batch normalisation

  Normalising each feature over the mini-batch, the learnable scale and shift, train vs eval mode, and its effect on a deep MLP.

• 17 Feb 2025 · Study

  Deep Learning · Weight initialisation

  Why initial weight scale matters, Xavier/Glorot for tanh and He/Kaiming for ReLU, and watching activations explode or collapse through 50 layers.

• 6 Feb 2025 · Study

  Deep Learning · Optimisers: SGD, momentum, Adam

  Mini-batch SGD, momentum, RMSProp-style scaling and Adam compared on an ill-conditioned valley, using torch.optim.

• 28 Jan 2025 · Study

  Deep Learning · Activation functions

  Sigmoid, tanh, ReLU, Leaky ReLU and GELU compared by shape and gradient, and why saturating activations make deep networks hard to train.

• 20 Jan 2025 · Study

  Deep Learning · Backpropagation

  Backpropagation as the chain rule applied layer by layer; a two-layer network trained on XOR in pure NumPy, plus a numerical gradient check.

• 9 Jan 2025 · Study

  Deep Learning · Multilayer perceptron

  Adding a hidden layer with a non-linear activation turns the perceptron into an MLP that can solve XOR; a forward pass written in NumPy.

• 1 Jan 2025 · Study

  Deep Learning · The perceptron

  Rosenblatt's perceptron from scratch: a weighted sum, a step function, and the update rule; it learns AND but can never learn XOR.

• 23 Dec 2024 · Study

  Machine Learning · Hyperparameter search

  Tuning with GridSearchCV and RandomizedSearchCV, nested evaluation on a held-out test set, and why random search often wins.

• 13 Dec 2024 · Study

  Machine Learning · Feature engineering

  Creating better inputs: ratios, date parts, binning, interactions, log transforms for skewed values, and target-free encodings.

All posts in the archive →