Study
NLP · Text preprocessing
Cleaning raw text with regex and spaCy: normalisation, lowercasing, stop words, lemmatisation, and why modern models need much less of this.
- Classic NLP pipelines normalise text heavily; Transformer models mostly want raw text and their own tokenizer.
- Lemmatisation maps words to dictionary forms (ran → run); stemming just chops suffixes.