AI updates
vLLM 0.4: automatic prefix caching
vLLM 0.4.0 added automatic prefix caching (reusing the KV cache for shared prompt prefixes such as long system prompts), JSON mode in its OpenAI-compatible server, and new models including DBRX and Command R+.
- Automatic prefix caching reuses the computed KV cache when many requests start with the same text, like a long system prompt.
- The release also moved forward on chunked prefill and speculative decoding.