Study
NLP · Attention (Bahdanau)
Additive attention lets the decoder look at every encoder state instead of one summary vector; the same reversal task with attention, and its alignment weights.
- Attention computes, at every decoding step, a weighted average of all encoder states, with weights the model learns.
- For string reversal, the learned attention weights come out close to an anti-diagonal: output step t mostly looks at input step L−1−t.