vLLM 0.4: automatic prefix caching
vLLM 0.4.0 added automatic prefix caching (reusing the KV cache for shared prompt prefixes such as long system prompts), JSON mode in its OpenAI-compatible server, and new models including DBRX and Command R+.
What happened
vLLM v0.4.0, an open-source LLM serving engine, was released.
The technical details
From the release notes:
- Automatic prefix caching: long shared prefixes, such as a system prompt, are cached across requests. Enable it with
--enable-prefix-caching. - OpenAI-compatible server:
json_objectoutput, a--use-delayflag to improve time-to-first-token under load, andmin_tokens. - progress on a chunked prefill scheduler and speculative decoding
- new models: Command R+, Qwen2 MoE, DBRX, XVerse, Jais, and the vision-language model LLaVA
- better AWS Inferentia (Neuron) support and a CMake-based build
Why it matters
Most chat and RAG apps send the same long instructions with every request. Without caching, the model recomputes attention keys and values for that text every time. Prefix caching skips that repeated work, which directly cuts latency and cost.
Sources
- vLLM v0.4.0 release notes — GitHub · vllm-project/vllm
- Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention — arXiv
This is a summary written from the linked public sources. Figures and numbers are as reported there.