LLMs · DPO · Sahaj's blog

Direct Preference Optimization skips the separate reward model and RL loop and trains on preference pairs with a simple classification-style loss; the loss implemented and checked on a toy policy.

RLHF needs a reward model and an RL loop, which is complex and finicky. DPO (Rafailov et al., 2023) shows that the same KL-constrained objective can be optimised directly from preference pairs:

$\mathcal{L}_{\text{DPO}} = -\log\sigma\!\Big(\beta\big[\log\tfrac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log\tfrac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\big]\Big)$

In words: raise the probability of the chosen answer and lower the rejected one, relative to a frozen reference model. The paper’s title says the policy is “secretly a reward model”: β × the log-ratio plays the role of the reward.

Toy check: a “policy” that chooses among 5 canned answers per prompt, with preference pairs saying answer 0 beats answer 4. DPO should shift probability toward answer 0, and a larger β should keep it closer to the reference.

import torch, torch.nn.functional as F
torch.manual_seed(0)

ref_logits = torch.zeros(5)                          # reference: uniform over 5 answers
def dpo(beta, steps=200):
    logits = ref_logits.clone().requires_grad_(True)
    opt = torch.optim.SGD([logits], lr=0.5)
    for _ in range(steps):
        logp, ref = logits.log_softmax(-1), ref_logits.log_softmax(-1)
        margin = beta * ((logp[0] - ref[0]) - (logp[4] - ref[4]))   # chosen=0, rejected=4
        loss = -F.logsigmoid(margin)
        opt.zero_grad(); loss.backward(); opt.step()
    p = logits.softmax(-1).detach()
    kl = (p * (p.log() - ref_logits.log_softmax(-1))).sum()
    return p, kl, loss.item()

for beta in (0.1, 0.5, 2.0):
    p, kl, l = dpo(beta)
    print(f"beta={beta}: probs {[round(v, 3) for v in p.tolist()]} | KL from ref {kl:.3f} | loss {l:.3f}")

Output:

beta=0.1: probs [0.946, 0.018, 0.018, 0.018, 0.0] | KL from ref 1.337 | loss 0.374
beta=0.5: probs [0.94, 0.02, 0.02, 0.02, 0.0] | KL from ref 1.315 | loss 0.021
beta=2.0: probs [0.626, 0.117, 0.117, 0.117, 0.022] | KL from ref 0.478 | loss 0.001

Answers 1–3 never appear in the preference pairs, so they’re only touched indirectly through the softmax. In this toy, a larger β reaches a low loss with a smaller move in log-ratio terms.

LLMs · DPO

Sources