Paper summary
Paper summary: The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
FineWeb is a 15-trillion-token open pre-training dataset built from 96 Common Crawl snapshots, with every curation choice ablated; its 1.3T-token educational subset, FineWeb-Edu, gives large gains on MMLU and ARC.
- The dataset comes with its recipe: deduplication and filtering choices are each tested by training small models.
- Filtering for educational content had a big effect on knowledge- and reasoning-heavy benchmarks.