Paper summary: ShowUI: One Vision-Language-Action Model for GUI Visual Agent

Problem

Most GUI agents are language-based. They rely on closed-source APIs and text meta-information like HTML or accessibility trees, so they don’t perceive the interface visually the way humans do.

Method

ShowUI is a vision-language-action model for the digital world, with:

1. UI-guided visual token selection: treat the screenshot as a UI connected graph, find redundant regions (large uniform areas), and drop those tokens in self-attention, which cuts compute.
2. Interleaved vision-language-action streaming: one format that handles visual–action history in navigation and multi-turn queries.
3. A curated, balanced GUI instruction-following dataset.

Key results

From the abstract: ShowUI, a lightweight 2B model trained on 256K samples, reaches 75.1% accuracy in zero-shot screenshot grounding. UI-guided token selection removes 33% of redundant visual tokens during training and speeds it up 1.4×. Navigation experiments on Mind2Web (web), AITW (mobile) and MiniWob (online) support the approach.

Limitations

There’s no dedicated limitations section in the extracted text. Working from screenshots alone means the model must infer structure that HTML would state explicitly.