Paper summary: GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Problem

Training LLMs needs memory for weights and optimizer states (Adam keeps two extra values per parameter). LoRA reduces this by training only a small low-rank update, but the abstract notes that LoRA-style methods usually underperform full-rank training. They limit the search to a low-rank subspace, change training dynamics, and may need a full-rank warm start.

Method

GaLore keeps full-parameter learning but makes the gradients low-rank. At each step, the gradient matrix is projected onto a low-rank subspace, where the optimizer keeps its states. The update is projected back and applied to all weights. The subspace is refreshed periodically, so over time training can still explore the full space.

Key results

From the abstract:

• up to 65.5% less memory in optimizer states, while keeping efficiency and performance for pre-training LLaMA 1B and 7B on C4 (up to 19.7B tokens) and for fine-tuning RoBERTa on GLUE
• 8-bit GaLore reduces optimizer memory by up to 82.5%
• the paper also reports pre-training a 7B model on a consumer GPU with 24 GB of memory, without model parallelism, checkpointing or offloading

Limitations

There’s no dedicated limitations section in the extracted text. The savings target optimizer states and gradients. Activations and weights still need memory as usual.