Sahaj.blogs
30 Apr 2025 · Paper summary

Paper summary: Reinforcement Learning for Reasoning in Large Language Models with One Training Example

RL with verifiable rewards on a single maths training example lifts Qwen2.5-Math-1.5B on MATH500 from 36.0% to 73.6%, matching training on a 1.2k-example dataset.

Problem

How much data does RL with verifiable rewards actually need to improve maths reasoning?

Method

Apply RLVR to a base model with just one training example (“1-shot RLVR”), and compare with the same training on a larger dataset that contains that example.

Key results

From the abstract, on Qwen2.5-Math-1.5B:

  • MATH500: 36.0% → 73.6% with one example, of which 8.6 points go beyond format correction
  • average across six maths benchmarks: 17.6% → 35.7% (7.0 points non-format gain)
  • this matches training on the 1.2k-example DeepScaleR subset (73.6%, 35.9%)
  • two examples slightly exceed it (74.8%, 36.6%)
  • similar gains appear for Qwen2.5-Math-7B, Llama3.2-3B-Instruct and others

Limitations

From the paper: larger models such as Qwen2.5-32B weren’t tried because of compute limits, and only the maths domain was studied, not coding or other verifiable domains.

#llm#reasoning#rl

Sources

  1. Reinforcement Learning for Reasoning in Large Language Models with One Training Example (arXiv:2504.20571) — arXiv
  2. Hugging Face Daily Papers, 2025-04-30 — Hugging Face
  3. Reinforcement Learning for Reasoning in Large Language Models with One Training Example (arXiv HTML full text) — arXiv
  4. Code repository (linked in the paper) — GitHub

This is a summary written from the linked public sources. Figures and numbers are as reported there.