Sahaj.blogs
11 Feb 2026 · Paper summary

Paper summary: OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

OPUS picks pre-training data at every iteration by projecting each candidate's optimizer-shaped update onto a target direction, at 4.7% extra compute; GPT-2 runs on 30B selected tokens beat full 200B-token training.

Figure from the paper "OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration"
Figure from Wang et al. (2026), licensed CC BY 4.0. Figure 1: OPUS outperforms random selection by an average of 2.2% accuracy across 10 benchmarks and achieves 8 [math] reduction in computation on GPT-XL using FineWeb dataset.

Problem

The authors call it the Data Wall: high-quality public text is running out, so pre-training must use better tokens rather than more. Existing selection methods are static heuristic filters (ignoring training dynamics) or dynamic but optimizer-agnostic gradient criteria.

Method

OPUS scores each candidate by projecting its effective update, as shaped by a modern optimizer, onto a target direction from a stable, in-distribution proxy. For scale, it uses a Ghost technique with CountSketch, and Boltzmann sampling for diversity. The overhead is only 4.7% extra compute.

Key results

From the abstract and Figure 1:

  • GPT-2 Large/XL on FineWeb and FineWeb-Edu with 30B selected tokens outperforms industrial baselines and even full 200B-token training
  • combined with static filters, it improves efficiency further, even with lower-quality data
  • continued pre-training of Qwen3-8B-Base on SciencePedia: 0.5B tokens beat full training on 3B
  • on GPT-XL/FineWeb, an average of 2.2% higher accuracy than random selection across 10 benchmarks

Limitations

There’s no dedicated limitations section in the extracted text. The pre-training experiments are at GPT-2 scale.

#llm#pre-training#data

Sources

  1. OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration (arXiv:2602.05400) — arXiv
  2. Hugging Face Daily Papers, 2026-02-11 — Hugging Face
  3. OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration (arXiv HTML full text) — arXiv

This is a summary written from the linked public sources. Figures and numbers are as reported there.