Paper summary
Paper summary: LLM in a flash: Efficient Large Language Model Inference with Limited Memory
Apple researchers run LLMs larger than the available DRAM by keeping weights in flash and loading only what is needed, with windowing and row–column bundling designed around how flash memory reads data.
- Two flash-aware tricks: reuse recently activated neurons (windowing), and store related rows and columns together so reads are larger and contiguous.
- The paper reports running models up to twice the size of DRAM, with 4–5× (CPU) and 20–25× (GPU) faster inference than naive loading.