#alignment

18 Feb 2026 · Study
LLMs · DPO
Direct Preference Optimization skips the separate reward model and RL loop and trains on preference pairs with a simple classification-style loss; the loss implemented and checked on a toy policy.
11 Feb 2026 · Study
LLMs · RLHF
Reinforcement learning from human feedback in three stages, with the reward-model step built from scratch: a Bradley–Terry model learning to prefer the better of two answers from pairwise labels.
9 Jan 2026 · Paper summary
Paper summary: GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
With several rewards (correctness, format, length…), GRPO's group normalisation can collapse different reward combinations into identical advantages; GDPO normalises each reward separately and trains more stably.
25 Jul 2025 · Paper summary
Paper summary: Group Sequence Policy Optimization
Group Sequence Policy Optimization computes importance ratios and clipping at the level of whole sequences rather than tokens, and is reported to be more stable and efficient than GRPO, especially for MoE models.
22 Nov 2024 · Paper summary
Paper summary: Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
Mixed Preference Optimization and the automatically built MMPR dataset improve chain-of-thought reasoning in multimodal LLMs (InternVL2-8B and 76B) beyond what supervised fine-tuning alone gives.
10 Sept 2024 · Paper summary
Paper summary: Towards a Unified View of Preference Learning for Large Language Models: A Survey
A survey that breaks every LLM preference-alignment method into four components, model, data, feedback and algorithm, to show how RLHF, DPO and their relatives connect.
16 Apr 2024 · Paper summary
Paper summary: Learn Your Reference Model for Real Good Alignment
Trust Region alignment (TR-DPO, TR-IPO, TR-KTO) updates the reference policy during offline alignment training, which reduces over-optimisation.
18 Jan 2024 · Paper summary
Paper summary: Self-Rewarding Language Models
A model judges its own responses with LLM-as-a-Judge prompting and trains on those rewards with iterative DPO, so both its instruction following and its judging improve over three rounds.
14 Aug 2023 · Paper summary
Paper summary: Self-Alignment with Instruction Backtranslation
Instruction backtranslation lets a language model label unlabelled web text with the instructions it answers, then pick the best pairs to fine-tune itself.