PyTorch 2.5: cuDNN SDPA backend, FlexAttention, regional compilation
PyTorch 2.5 added a cuDNN backend for scaled dot-product attention (speedups by default on H100-class GPUs), regional compilation to cut torch.compile cold-start time, and a FlexAttention prototype.
What happened
PyTorch 2.5 was released, with 4,095 commits from 504 contributors since 2.4.
The technical details
From the release notes:
- cuDNN backend for SDPA (beta): speedups by default for SDPA users on H100s or newer
- regional compilation (beta): compile a repeated
nn.Module, such as a Transformer layer, without recompilations, to cuttorch.compilecold-start time - TorchInductor CPU backend improvements: FP16, CPP wrapper, AOT-Inductor mode, max-autotune
- prototypes: FlexAttention, Compiled Autograd, Flight Recorder, max-autotune on CPU, and TorchInductor on Windows
Why it matters
Attention keeps getting new variants (sliding windows, soft-capping, masks), and each used to need a custom CUDA kernel. FlexAttention aims to let researchers express a variant in PyTorch and still get a fused, fast kernel. Regional compilation addresses a practical annoyance: torch.compile taking a long time to start on big models.
Sources
- PyTorch 2.5 release notes — GitHub · pytorch/pytorch
This is a summary written from the linked public sources. Figures and numbers are as reported there.