Paper summary: rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

Problem

Strong maths reasoning seemed to need very large models, or distillation from them. Can small language models get there on their own?

Method

rStar-Math has a policy SLM search step by step through solutions with Monte Carlo Tree Search, guided by an SLM-based process reward model. Three ideas make the two small models trainable:

1. Code-augmented CoT synthesis: extensive MCTS rollouts generate step-by-step reasoning in which each step includes code that can be executed, giving verified trajectories for training the policy.
2. A process preference model (PPM), trained without naive per-step score labels.
3. Self-evolution: policy and PPM are built from scratch and improved over several rounds.

Key results

From the abstract, after 4 rounds of self-evolution with millions of synthesised solutions for 747k problems:

• MATH: Qwen2.5-Math-7B goes from 58.8% to 90.0%, and Phi3-mini-3.8B from 41.4% to 86.4%, above o1-preview by 4.5 and 0.9 points
• AIME: an average of 53.3% (8 of 15) problems solved

All of this without distillation from a superior model.

Limitations

There’s no dedicated limitations section in the extracted text. MCTS at test time means many model calls per problem, so the approach trades inference compute for accuracy.