#optimisation

• 6 Feb 2025 · Study

  Deep Learning · Optimisers: SGD, momentum, Adam

  Mini-batch SGD, momentum, RMSProp-style scaling and Adam compared on an ill-conditioned valley, using torch.optim.

• 19 Aug 2024 · Study

  Machine Learning · Gradient descent

  Fitting linear regression by gradient descent from scratch: the gradient of MSE, the learning rate, and what happens when it is too small or too big.

• 8 Sept 2023 · Paper summary

  Paper summary: Large Language Models as Optimizers

  OPRO describes an optimisation task in natural language and lets an LLM propose new solutions from a history of past solutions and scores, used mainly to optimise prompts.