Paper summary: YuLan-Mini: An Open Data-efficient Language Model

Problem

Pre-training LLMs is expensive and the full process is complex, which limits who can do it, especially academic groups.

Method

YuLan-Mini (2.42B parameters) focuses on training efficacy, with three contributions:

1. an elaborate data pipeline that combines cleaning with data-schedule strategies
2. a robust optimisation method to reduce training instability
3. an annealing phase with targeted data selection and long-context training

Key results

From the abstract: top-tier performance among models of similar size, and, trained on 1.08T tokens, performance comparable to industry models that use significantly more data. Full details of the data composition for each training phase are released to support reproduction.

Limitations

There’s no dedicated limitations section in the extracted text. It’s a base model of modest size, so it isn’t comparable with large frontier models.