Paper summary: FlowRL: Matching Reward Distributions for LLM Reasoning

Problem

PPO and GRPO maximise reward. According to the abstract, this tends to over-optimise dominant reward signals and neglect less frequent but valid reasoning paths, which reduces diversity.

Method

FlowRL matches the full reward distribution instead:

1. turn scalar rewards into a normalised target distribution using a learnable partition function
2. minimise the reverse KL divergence between the policy and that target

Implemented as flow-balanced optimisation, this encourages diverse exploration.

Key results

From the abstract: on maths benchmarks, an average improvement of 10.0% over GRPO and 5.1% over PPO, and consistently better results on code reasoning.

Limitations

There’s no dedicated limitations section in the extracted text. Results cover maths and code, where rewards are verifiable.