Study
Transformers · Self-attention
Scaled dot-product self-attention from scratch: queries, keys, values, the √d scaling, softmax weights, and a causal mask, checked against PyTorch's built-in function.
- Each token makes a query, compares it with every key, and takes a softmax-weighted average of the values.
- Dividing by √d keeps the dot products from growing with dimension and saturating the softmax.