Paper summary: Self-Alignment with Instruction Backtranslation

Problem

Instruction tuning needs many (instruction, response) pairs. Human-written pairs are expensive, and distilling them from a stronger proprietary model brings its own dependency. Meanwhile the web is full of high-quality human-written responses (guides, explanations) that have no instruction attached.

Method

The name borrows from backtranslation in machine translation:

1. Fine-tune a seed model on a small amount of human-written instruction data.
2. Self-augmentation: for each web document, have the model generate the instruction that the document would answer.
3. Self-curation: have the model itself rate the candidate pairs, and keep only the high-quality ones.
4. Fine-tune a stronger model on the curated pairs and repeat. The paper uses two iterations.

Key results

The authors report that fine-tuning LLaMa with two iterations gives a model (called Humback) that outperforms all other LLaMa-based models on the Alpaca leaderboard that don’t rely on distillation data. They describe this as effective self-alignment.

Limitations

The paper doesn’t include a separate limitations section in the HTML version. By design, the method depends on the web corpus it starts from and on the model’s own quality ratings during self-curation. The headline result in the abstract is measured on the Alpaca leaderboard.