#tokenisation
- 9 May 2025 · Study
NLP · Tokenisation
Word, character and subword tokenisation compared, out-of-vocabulary words, and what a real BERT WordPiece tokenizer does to unusual words.
Word, character and subword tokenisation compared, out-of-vocabulary words, and what a real BERT WordPiece tokenizer does to unusual words.