Study
Transformers · BERT
BERT as a bidirectional encoder pretrained with masked language modelling; running fill-mask, looking at [CLS] and [SEP], and seeing contextual embeddings change with meaning.
- Masked language modelling lets BERT use context on both sides of a word, unlike left-to-right models.
- The same word gets different vectors in different sentences: "bank" of a river vs a bank account.