#training

• 25 Mar 2025 · Study

  Deep Learning · PyTorch training loop

  The standard PyTorch loop with Dataset, DataLoader, nn.Module, a loss, an optimiser and an evaluation pass, applied to scikit-learn's handwritten digits.

• 7 Mar 2024 · Paper summary

  Paper summary: GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

  GaLore projects gradients into a low-rank subspace for the optimizer state while still updating all weights, cutting optimizer memory by up to 65.5% (82.5% in 8-bit) without LoRA's accuracy trade-off.