Study
Transformers · Positional encoding
Why attention needs position information, the sinusoidal encoding from the original paper, learned position embeddings, and the relative-position idea behind RoPE.
- Without positional information, self-attention gives the same output for any shuffling of the input tokens.
- Sinusoids at different frequencies give every position a unique pattern, and nearby positions get similar ones.