Paper summary: Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

Problem

Reasoning models usually spend more test-time compute by generating more tokens: long chains of thought. That needs suitable training data and long context windows, and it forces all reasoning through words.

Method

The authors build a model with a recurrent block that can be iterated any number of times. At test time it can be unrolled to arbitrary depth, so it “thinks” longer in latent space before producing a token. According to the abstract, this:

• needs no specialised training data
• works with small context windows
• can capture kinds of reasoning that are hard to express in words

They scale a proof of concept to 3.5B parameters trained on 800B tokens.

Key results

The model improves on reasoning benchmarks as it’s given more iterations, sometimes dramatically, up to a compute load the authors equate to a 50B-parameter model.

Limitations

There’s no dedicated limitations section in the extracted text. It’s described as a proof of concept at 3.5B scale.