Paper summary: Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

Problem

RL with verifiable rewards (RLVR), as in DeepSeek-R1 (Jan 2025 on this blog), clearly improves reasoning scores. A common belief is that it lets models discover new reasoning strategies beyond their base model. Is that true?

Method

The authors probe the boundary of what models can solve, using pass@k at large k: with many attempts, can the model solve the problem at least once? They compare base and RLVR-trained models across model families, RL algorithms, and maths, coding and visual reasoning benchmarks, plus coverage and perplexity analyses.

Key results

From the abstract:

• RLVR models beat their base models at small k (e.g. k = 1), but base models reach higher pass@k at large k
• the reasoning abilities seen after RLVR originate from and are bounded by the base model
• six popular RLVR algorithms perform similarly and remain far from fully using the base model’s potential
• distillation from a stronger teacher, by contrast, can add genuinely new reasoning patterns

Limitations

The paper frames its conclusions about current RLVR methods. The authors suggest the limitation may come from weak exploration in the vast language space, and point to continual scaling and multi-turn agent–environment interaction as directions.