Paper summary: Reinforcement Pre-Training

Problem

RL for reasoning usually needs domain-specific problems with checkable answers (maths, code). Meanwhile, huge amounts of general text are only used for plain next-token prediction.

Method

Reinforcement Pre-Training (RPT) turns next-token prediction into a reasoning task trained with RL. For a given context, the model reasons and then predicts the next token, and it receives a verifiable reward if the prediction is correct. Any text corpus becomes a source of RL training signal.

Key results

From the abstract:

• RPT significantly improves next-token prediction accuracy
• it gives a stronger foundation for later reinforcement fine-tuning
• scaling curves show accuracy improving consistently with more training compute

Limitations

There’s no dedicated limitations section in the extracted text. Reasoning before every token is far more expensive than standard next-token training.