Sahaj.blogs
9 Jan 2026 · Paper summary

Paper summary: GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization

With several rewards (correctness, format, length…), GRPO's group normalisation can collapse different reward combinations into identical advantages; GDPO normalises each reward separately and trains more stably.

Figure from the paper "GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization"
Figure from Liu et al. (2026), licensed CC BY 4.0. (a) An overview of our proposed GDPO

Problem

RL pipelines now use multiple rewards: correctness, format, length, and other preferences. Most simply apply GRPO to the combined reward. The authors show that normalising combined rewards this way can make different reward combinations collapse into identical advantage values. That reduces the signal’s resolution, causes suboptimal convergence, and sometimes leads to early training failure.

Method

GDPO (Group reward-Decoupled Normalization Policy Optimization) normalises each reward separately before combining them, which keeps their relative differences and makes multi-reward optimisation more accurate and more stable.

Key results

From the abstract: across tool calling, maths reasoning and coding reasoning, measured by correctness (accuracy, bug ratio) and constraint adherence (format, length), GDPO consistently outperforms GRPO.

Limitations

There’s no dedicated limitations section in the extracted text. Experiments cover three task families.

#llm#rl#alignment

Sources

  1. GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization (arXiv:2601.05242) — arXiv
  2. Hugging Face Daily Papers, 2026-01-09 — Hugging Face
  3. GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization (arXiv HTML full text) — arXiv
  4. Code repository (linked in the paper) — GitHub

This is a summary written from the linked public sources. Figures and numbers are as reported there.