Transformers · BERT

BERT (Devlin et al., 2018) is an encoder-only Transformer, pre-trained with two tasks:

1. Masked language modelling (MLM): hide about 15% of the tokens and predict them from the context on both sides.
2. Next sentence prediction: did sentence B really follow sentence A?

Special tokens: [CLS] at the start (its vector is used for classification) and [SEP] between and after sentences. After pre-training, BERT is fine-tuned for each task with a small head on top.

The difference from word2vec and GloVe is that BERT’s vectors are contextual. I checked this with “bank” in two senses, three sentences each. My first try compared just one pair, and it came out the wrong way round: one sentence pair is too noisy. Averaging over all pairs gives a much clearer picture.

import torch
from transformers import pipeline, AutoTokenizer, AutoModel

fill = pipeline("fill-mask", model="bert-base-uncased")
for r in fill("The students at PDEU study [MASK] engineering.", top_k=4):
    print(f"{r['token_str']:>12}  {r['score']:.3f}")

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
bert = AutoModel.from_pretrained("bert-base-uncased", output_hidden_states=True).eval()
print(tok.convert_ids_to_tokens(tok("I sat by the river.", "It was calm.")["input_ids"]))

def vec(sentence, layer, word="bank"):
    enc = tok(sentence, return_tensors="pt")
    with torch.no_grad():
        h = bert(**enc).hidden_states[layer][0]
    return h[enc.input_ids[0].tolist().index(tok.convert_tokens_to_ids(word))]

river = ["we sat on the bank of the river", "fish swam near the muddy river bank", "the boat drifted toward the grassy bank of the stream"]
money = ["i deposited cash at the bank today", "the bank approved my loan application", "she opened a savings account at the bank"]
cos = torch.nn.functional.cosine_similarity
for layer in (4, 12):
    R, M = [vec(s, layer) for s in river], [vec(s, layer) for s in money]
    same = [cos(a, b, dim=0).item() for grp in (R, M) for i, a in enumerate(grp) for b in grp[i + 1:]]
    diff = [cos(a, b, dim=0).item() for a in R for b in M]
    print(f"layer {layer:>2}: avg similarity same sense {sum(same) / len(same):.3f} | different sense {sum(diff) / len(diff):.3f}")

Output:

  mechanical  0.371
       civil  0.120
    computer  0.099
  electrical  0.096
['[CLS]', 'i', 'sat', 'by', 'the', 'river', '.', '[SEP]', 'it', 'was', 'calm', '.', '[SEP]']
layer  4: avg similarity same sense 0.764 | different sense 0.479
layer 12: avg similarity same sense 0.590 | different sense 0.414