Paper summary: SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

Problem

Supervised fine-tuning (SFT) and reinforcement learning (RL) are both standard post-training steps, but it’s unclear how each affects generalisation versus memorisation.

Method

Two test beds with controlled variations:

• GeneralPoints: an arithmetic reasoning card game, with variants in the rules (text) and in how the cards look (visual)
• V-IRL: a real-world navigation environment

Models trained with SFT or RL are evaluated on unseen variants, out of distribution.

Key results

From the abstract and Figure 1:

• RL, especially with an outcome-based reward, generalises across both rule-based textual and visual variants
• SFT tends to memorise training data and struggles out of distribution
• RL also improves the model’s underlying visual recognition
• SFT still helps, by stabilising the output format so that later RL can work

Limitations

From the paper: two challenges were unresolved. One is that SFT failed to reach in-distribution performance comparable to RL on the visual-language GeneralPoints variant, even after extra hyperparameter tuning to rule out variance.