Paper summary: Orca: The World is in Your Mind

Problem

Models usually learn one kind of prediction: next token, next frame or next action. The authors want one model of how the world changes, usable for understanding, prediction and action.

Method

Orca learns a unified world latent space through Next-State-Prediction, with two kinds of learning:

• “unconscious” learning: dense, natural state transitions from continuous video
• “conscious” learning: sparse, meaningful transitions described by language events and VQA

Pre-training data: 125K hours of video and 160M event annotations. After pre-training, the backbone is frozen, and only small modality-specific decoders are trained for three readouts: text, image prediction and embodied action.

Key results

From the abstract: the approach scales, a stronger world latent gives stronger downstream readouts, and Orca outperforms similar-sized specialised baselines.

Limitations

There’s no dedicated limitations section in the extracted text. The authors call it an “initial instantiation” of a general world model.