Paper summary: Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing

Problem

RL post-training (as in DeepSeek-R1-Zero) needs heavy parallel inference, which brings latency, memory and reliability problems, and growing cost.

Method

SAPO is fully decentralised and asynchronous. Each node in a network of heterogeneous computers manages its own policy model(s) and shares rollouts with others. Nodes can also run alone. There are no explicit assumptions about latency, model homogeneity or hardware. By sampling rollouts shared across the network, insights (“aha moments”) can propagate between nodes and bootstrap learning.

Key results

From the abstract: cumulative reward gains of up to 94% in controlled experiments. The authors also report insights from an open-source demo run on thousands of community nodes with diverse hardware and models.

Limitations

There’s no dedicated limitations section in the extracted text. The large-scale run was a community demo, so it’s less controlled than the main experiments.