LLMs · QLoRA

LoRA cuts the number of trainable parameters, but the frozen base model still has to sit in memory. QLoRA (Dettmers et al., 2023) shrinks that too:

• the frozen base weights are stored in 4-bit NormalFloat (NF4)
• blockwise quantisation: each block of 64 weights has its own scale (absmax)
• double quantisation: the scales themselves are quantised
• paged optimizers handle memory spikes
• LoRA adapters train in 16-bit on top, with the 4-bit weights dequantised on the fly

The paper reports fine-tuning a 65B model on a single 48GB GPU while matching 16-bit fine-tuning performance on their benchmarks.

Why “NormalFloat”? Pretrained weights are roughly normally distributed. NF4 places its 16 levels at quantiles of a normal distribution, so each level is used about equally. A uniform 4-bit grid wastes levels on rare large values. I built both by hand, with blockwise absmax scaling, and compared reconstruction error on normally distributed weights.

import torch
from scipy.stats import norm
torch.manual_seed(0)

# 16 NF4-style levels: normal quantiles, normalised to [-1, 1] (an approximation of the paper's construction)
q = torch.tensor(norm.ppf(torch.linspace(0.02, 0.98, 16).numpy()), dtype=torch.float32)
nf4 = q / q.abs().max()
uniform4 = torch.linspace(-1, 1, 16)

def quantise(w, levels, block=64):
    w = w.reshape(-1, block)
    scale = w.abs().max(dim=1, keepdim=True).values          # absmax per block
    idx = ((w / scale).unsqueeze(-1) - levels).abs().argmin(-1)   # nearest level -> 4-bit code
    return (levels[idx] * scale).reshape(-1), idx

w = torch.randn(4096 * 64) * 0.02
for name, lv in [("uniform 4-bit", uniform4), ("NF4-style", nf4)]:
    deq, codes = quantise(w, lv)
    use = torch.bincount(codes.reshape(-1), minlength=16).float()
    print(f"{name:>13}: mean abs error {(deq - w).abs().mean():.2e} | least-used level gets {use.min() / use.sum():.2%} of weights")

params = 7e9
for bits in (16, 8, 4):
    print(f"7B model weights at {bits:>2}-bit: {params * bits / 8 / 1e9:5.1f} GB")

Output:

uniform 4-bit: mean abs error 1.71e-03 | least-used level gets 1.13% of weights
    NF4-style: mean abs error 1.46e-03 | least-used level gets 1.98% of weights
7B model weights at 16-bit:  14.0 GB
7B model weights at  8-bit:   7.0 GB
7B model weights at  4-bit:   3.5 GB