Study
LLMs · Fine-tuning with LoRA
LoRA freezes the pretrained weights and learns a low-rank update BA; a LoRA layer written from scratch, then PEFT applied to GPT-2 to count how few parameters are trained.
- LoRA learns ΔW = B·A with a small rank r, so a d×d update needs only 2·d·r trainable numbers.
- B starts at zero, so the model is unchanged at the start of fine-tuning.