#pre-training
- 11 Feb 2026 · Paper summary
Paper summary: OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
OPUS picks pre-training data at every iteration by projecting each candidate's optimizer-shaped update onto a target direction, at 4.7% extra compute; GPT-2 runs on 30B selected tokens beat full 200B-token training.
- 1 Jan 2026 · Study
LLMs · Pre-training
Pre-training as next-token prediction on lots of text, shown end to end with a tiny character-level Transformer trained on a few paragraphs, before and after.
- 30 Oct 2025 · Paper summary
Paper summary: Scaling Latent Reasoning via Looped Language Models
Ouro builds latent reasoning into pre-training with looped language models (iterating in latent space with a learned depth), and its 1.4B and 2.6B models are reported to match up to 12B models.
- 10 Jun 2025 · Paper summary
Paper summary: Reinforcement Pre-Training
Reinforcement Pre-Training reframes next-token prediction as a reasoning task: the model reasons before predicting the next token and gets a verifiable reward when it's right, so ordinary text becomes RL training data.
- 3 Jan 2025 · Paper summary
Paper summary: 2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining
The authors turn 22,000 class hours of instructional videos into an interleaved image–text "textbook" corpus for vision-language pre-training, with denser knowledge than web-crawled interleaved data.
- 27 Dec 2024 · Paper summary
Paper summary: YuLan-Mini: An Open Data-efficient Language Model
YuLan-Mini is a 2.42B-parameter base model trained on 1.08T tokens that the authors report matches models trained on much more data, with full details of the data and training released.
- 26 Jun 2024 · Paper summary
Paper summary: The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
FineWeb is a 15-trillion-token open pre-training dataset built from 96 Common Crawl snapshots, with every curation choice ablated; its 1.3T-token educational subset, FineWeb-Edu, gives large gains on MMLU and ARC.
- 21 Jun 2024 · Paper summary
Paper summary: Instruction Pre-Training: Language Models are Supervised Multitask Learners
Instruction Pre-Training augments raw pre-training text with synthesised instruction–response pairs (200M of them across 40+ task types) and reports stronger base models and better gains from later instruction tuning.