Study
LLMs · Tokenizers: BPE
Byte-pair encoding from scratch: start from characters, repeatedly merge the most frequent adjacent pair, and use the learned merges to tokenise new words, compared with GPT-2's real tokenizer.
- BPE learns its vocabulary by repeatedly merging the most frequent adjacent pair of symbols.
- Frequent words become single tokens; rare words are split into known pieces.