Paper summary: Instruction Pre-Training: Language Models are Supervised Multitask Learners

Problem

Pre-training is unsupervised: predict the next token of raw text. Supervised multitask learning (instruction tuning) improves generalisation, but it usually happens only after pre-training. Can supervision be brought into pre-training itself, at scale?

Method

Instruction Pre-Training adds instruction–response pairs to the raw corpus. An instruction synthesizer, built on open-source models, reads a raw text and generates related tasks with answers. The LM is pre-trained on the raw text plus these pairs. The authors synthesise 200M pairs across 40+ task categories.

Key results

From the abstract:

• pre-training from scratch: consistently better base models, which also gain more from later instruction tuning
• continual pre-training: Llama3-8B becomes comparable to, or better than, Llama3-70B on their evaluations

Limitations

From the paper’s own section:

• synthetic data can introduce hallucinations; their analysis puts the accuracy of the instruction-augmented corpora at about 70%
• the pre-training scale is far below the trillions of tokens used for frontier models