Paper summary: Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning

Problem

Improving models on hard, verifiable tasks usually needs good training data, which may not exist. Sometimes all you have is a binary signal: did it succeed or not?

Method

A two-stage loop:

1. When the model fails a task, it writes a self-reflective commentary analysing its previous attempt.
2. It gets another attempt with that reflection in context.
3. If the retry succeeds, the tokens of the self-reflection are rewarded with RL.

The model is optimised to write useful reflections rather than to memorise answers.

Key results

From the abstract: substantial gains across model architectures, up to 34.7% on math equation writing and 18.1% on function calling. Smaller fine-tuned models (1.5B–7B) outperform models in the same family that are 10× larger.

Limitations

From the paper:

• a binary success/fail validator isn’t easy to define for every task
• the method doesn’t work for every model and task: the model needs basic ability to do the task, reflect and learn (e.g. Llama3.2-3B Instruct couldn’t learn to self-correct on function calling)