#serving

• 15 May 2025 · AI updates

  vLLM 0.9: PyTorch 2.7, Blackwell kernels, large-scale inference

  vLLM 0.9.0 moved to PyTorch 2.7 and CUDA 12.8, added initial optimised kernels for NVIDIA Blackwell, and started data-parallel, expert-parallel and prefill/decode disaggregation support for large-scale inference.

• 30 Mar 2024 · AI updates

  vLLM 0.4: automatic prefix caching

  vLLM 0.4.0 added automatic prefix caching (reusing the KV cache for shared prompt prefixes such as long system prompts), JSON mode in its OpenAI-compatible server, and new models including DBRX and Command R+.