Paper summary: LLM in a flash: Efficient Large Language Model Inference with Limited Memory

Problem

On phones and laptops, DRAM is the bottleneck. Many LLMs simply don’t fit. Flash storage is much larger but slower, and it performs best with large sequential reads.

Method

The authors build an inference cost model for flash memory, which points to two goals: transfer less data, and read it in bigger contiguous chunks. Two techniques follow:

1. Windowing: keep recently activated feed-forward neurons in DRAM and load only the newly needed ones, exploiting the sparsity of activations.
2. Row–column bundling: store the related row and column of each neuron’s weights together, so one read fetches both.

Key results

From the abstract: these methods let models run that are up to twice the size of available DRAM, with 4–5× (CPU) and 20–25× (GPU) faster inference than naive loading.

Limitations

The paper’s own limitations section says:

• power consumption and thermal limits still need systematic study for on-device use
• the work covers single-batch inference only; speculative decoding is just a preliminary result
• the proof of concept assumes about half the model fits in memory