#bpe
- 14 Jan 2026 · Study
LLMs · Tokenizers: BPE
Byte-pair encoding from scratch: start from characters, repeatedly merge the most frequent adjacent pair, and use the learned merges to tokenise new words, compared with GPT-2's real tokenizer.
Byte-pair encoding from scratch: start from characters, repeatedly merge the most frequent adjacent pair, and use the learned merges to tokenise new words, compared with GPT-2's real tokenizer.