Paper summary: The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

Problem

The pre-training data of leading open-weight models like Llama 3 and Mixtral isn’t public, and little is known about how it was built. Data quality and size matter enormously, so this is a big gap in public knowledge.

Method

Hugging Face’s FineWeb is a 15-trillion-token dataset from 96 Common Crawl snapshots. The paper documents and ablates every design choice, especially deduplication and filtering strategies, by training models on each variant and comparing them. It also introduces FineWeb-Edu, a 1.3-trillion-token subset filtered for educational text.

Key results

From the abstract:

• FineWeb produces better-performing LLMs than other open pre-training datasets
• models trained on FineWeb-Edu are dramatically better on knowledge- and reasoning-heavy benchmarks such as MMLU and ARC
• the curation code and all ablation models are released

Limitations

There’s no dedicated limitations section in the extracted text. The pipeline keeps English pages (it filters out non-English ones), and “quality” is defined by the authors’ filters and benchmark-based ablations.