Paper summary: Learning to Model the World with Language

Problem

Most instruction-following agents learn a direct mapping from an instruction (“pick up the blue ball”) to actions. Real language is much broader than commands: “this button turns on the TV”, “I put the bowls away”, feedback, game rules. Agents trained the direct way get worse as the language becomes more varied.

Method

The paper’s key idea is to treat language as one more signal for predicting the future. Dynalang learns a multimodal world model that, at each step, takes in an image and a text token and learns to predict future representations of both, plus rewards. The policy is trained on imagined rollouts inside that learned model, rather than only on real interaction. Because the world model is generative, it can also be pretrained on text-only data and can generate language grounded in the environment.

Key results

According to the authors, Dynalang learns to use environment descriptions, game rules and instructions across four environments, from simple games to navigation in photorealistic home scans. It outperforms model-free baselines, which struggle as language becomes more diverse.

Limitations

The paper doesn’t have a dedicated limitations section in its HTML version. Its conclusion frames the work as one step toward multimodal agents. The environments are simulated, and the results are on the benchmarks the authors built or chose.