Paper summary: Towards a Unified View of Preference Learning for Large Language Models: A Survey

A survey that breaks every LLM preference-alignment method into four components, model, data, feedback and algorithm, to show how RLHF, DPO and their relatives connect.

Figure from the paper "Towards a Unified View of Preference Learning for Large Language Models: A Survey"

Figure from Gao et al. (2024), licensed CC0 1.0. Figure 1 : A unified view and an illustrative example of preference learning for LLMs.

Problem

Aligning LLM outputs with human preferences is key to their usefulness, and research on it spans many areas. The relationships between methods (RLHF, DPO and many variants) are under-explored, which makes the field hard to navigate.

Method

The survey decomposes every preference-learning strategy into four components:

Model: the policy being trained
Data: the preference data and how it’s collected (the survey distinguishes on-policy and off-policy collection)
Feedback: how preference signals or rewards are obtained
Algorithm: how the model is updated from that feedback

The paper’s Figure 1 shows this unified view with an example.

Key results

The contribution is the framework, which links existing algorithms and points to new combinations. A companion reading list is maintained on GitHub.

Limitations

As a survey, it reflects the literature up to its writing in late 2024. The four-component split is a lens for comparing methods, not a performance result.