Paper summary: Self-Rewarding Language Models

Problem

Standard RLHF trains a separate reward model on human preferences and then freezes it. That reward model is limited by human labelling and can’t improve as the LLM improves.

Method

In a self-rewarding loop, the language model plays both roles:

1. It generates several candidate responses to new prompts.
2. Prompted as a judge (LLM-as-a-Judge), it scores its own candidates.
3. The best/worst pairs become preference data for DPO training.
4. Repeat with the improved model (Iterative DPO).

Key results

Fine-tuning Llama 2 70B for three iterations gives a model that, per the abstract, outperforms many systems on the AlpacaEval 2.0 leaderboard, including Claude 2, Gemini Pro and GPT-4 0613. The authors also report that the model’s reward-modelling ability improves along with its instruction following.

Limitations

From the paper: the results are preliminary. Only three iterations in one setting were run, and safety evaluation and the limits of iterative training are left open. They also observed longer generations over iterations. Length is known to correlate with judged quality, which needs deeper study.