#dpo

• 18 Feb 2026 · Study

  LLMs · DPO

  Direct Preference Optimization skips the separate reward model and RL loop and trains on preference pairs with a simple classification-style loss; the loss implemented and checked on a toy policy.

• 16 Apr 2024 · Paper summary

  Paper summary: Learn Your Reference Model for Real Good Alignment

  Trust Region alignment (TR-DPO, TR-IPO, TR-KTO) updates the reference policy during offline alignment training, which reduces over-optimisation.