Study
NLP · Tokenisation
Word, character and subword tokenisation compared, out-of-vocabulary words, and what a real BERT WordPiece tokenizer does to unusual words.
- Subword tokenisation splits rare words into known pieces, so there are no out-of-vocabulary words.
- Word-level vocabularies explode in size; character-level sequences become very long.