Paper summary: SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model

Problem

Large models are costly to run, so small ones matter for on-device and low-resource use. The question is how to make a 1.7B model as strong as possible.

Method

Hugging Face’s SmolLM2 is overtrained on about 11 trillion tokens, far beyond the compute-optimal amount for its size (the Chinchilla post on this blog explains that trade-off). Training runs in multiple stages that mix web text with specialised maths, code and instruction-following data. After each stage, the mixing rates are refined manually based on performance, guided by small-scale ablations. Where existing data was too small or low quality, the authors built new datasets: FineMath, Stack-Edu and SmolTalk.

Key results

From the abstract: SmolLM2 outperforms other recent small LMs, including Qwen2.5-1.5B and Llama3.2-1B. The model and all the datasets are released.

Limitations

There’s no dedicated limitations section in the extracted text. The stage-wise mixing involved manual decisions, which makes the recipe harder to transfer automatically.