Paper summary: Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds

Problem

Agents that act in rich 3D worlds for hours, in real time, following natural-language instructions, are far beyond most benchmark agents, which work in short, simplified tasks.

Method

Lumine is presented as the first open recipe for such agents. It uses a human-like interaction loop powered by a vision-language model:

• perceives raw pixels at 5 Hz
• outputs precise keyboard and mouse actions at 30 Hz
• invokes reasoning only when needed, to stay real-time

It’s trained in Genshin Impact.

Key results

From the abstract:

• completes the entire five-hour Mondstadt main storyline with human-level efficiency
• follows instructions across exploration, 2D GUI manipulation, collection, combat, puzzle-solving and NPC interaction
• zero-shot to other games: 100-minute missions in Wuthering Waves and the full five-hour first chapter of Honkai: Star Rail

Limitations

There’s no dedicated limitations section in the extracted text. The demonstrations are in a family of commercial games with similar interaction styles.