Paper summary: Vision language models are blind: Failing to translate detailed visual features into words

BlindTest's seven simple visual tasks, such as whether two circles overlap or how many times two lines cross, trip up strong vision-language models: four state-of-the-art VLMs averaged 58.07%.

Figure from the paper "Vision language models are blind: Failing to translate detailed visual features into words"

Figure from Rahmanzadehgervi et al. (2024), licensed CC BY 4.0. Figure 1 : VLMs fail on the simple tasks of BlindTest .

Problem

GPT-4o, Gemini 1.5 Pro and similar models score highly on vision benchmarks. But do they actually see precise spatial details?

Method

BlindTest is a suite of 7 very simple tasks that are easy for people, such as:

do two circles overlap?
how many times do two lines intersect?
which letter in a word is circled?
how many circles are in an Olympic-like logo?

Images vary in resolution and line width. Linear probes on the vision encoders then check whether the needed information is present before the language model sees it.

Key results

From the abstract:

four state-of-the-art VLMs average 58.07% accuracy
the best, Claude 3.5 Sonnet, reaches 77.84%, far from the expected human 100%
models do well when shapes are spread apart, and fail when they overlap or are close
linear probes show the vision encoders contain enough information to solve BlindTest, so the failure is in how it’s translated into language

Limitations

There’s no dedicated limitations section in the extracted text. The tasks are synthetic and geometric by design. They isolate one weakness and aren’t a general measure of visual ability.

Paper summary: Vision language models are blind: Failing to translate detailed visual features into words

Problem

Method

Key results

Limitations

Sources