Study
Transformers · Encoder and decoder
How the encoder layer, the decoder layer and cross-attention fit together, with residual connections and layer norm, and the three families they led to: encoder-only, decoder-only and encoder–decoder.
- The decoder has three sublayers: masked self-attention, cross-attention to the encoder output, and a feed-forward network.
- BERT keeps only the encoder, GPT only the decoder, and T5 keeps both.