Paper summary
Paper summary: YuLan-Mini: An Open Data-efficient Language Model
YuLan-Mini is a 2.42B-parameter base model trained on 1.08T tokens that the authors report matches models trained on much more data, with full details of the data and training released.
- Three levers: a data pipeline with cleaning and scheduling, an optimisation recipe against instability, and annealing with targeted data and long-context training.
- 1.08T training tokens is far fewer than many comparable models use.