Paper summary: Learn Your Reference Model for Real Good Alignment

Problem

Offline alignment methods like DPO don’t need a separate reward model, but they can still over-optimise. As the trained model moves far from the reference policy, sample quality drops.

Method

The Trust Region (TR) family (TR-DPO, TR-IPO, TR-KTO) updates the reference policy during training, instead of keeping it frozen at the starting model. The reference can be blended softly toward the current policy, or replaced periodically. The model’s deviation is then measured against a moving anchor.

Key results

From the abstract:

• toy examples show reduced over-optimisation
• in side-by-side comparisons on helpful and harmless dialogue and summarisation, TR methods surpass conventional ones
• with Llama3, they report significant improvements on AlpacaEval 2 and Arena-Hard

Limitations

There’s no dedicated limitations section in the extracted text. The update schedule for the reference policy adds new hyperparameters to tune.