Transformers · Vision Transformer (ViT)

ViT (Dosovitskiy et al., 2020) applies an almost unchanged Transformer encoder to images:

1. Cut the image into 16×16 patches. A 224×224 image gives 14 × 14 = 196 patches.
2. Flatten each patch (16 × 16 × 3 = 768 numbers) and linearly project it to a token. A stride-16 convolution does exactly this.
3. Prepend a learnable [CLS] token and add position embeddings.
4. Run a standard Transformer encoder, and classify from the [CLS] output.

The paper found that ViT needs large pre-training data to beat CNNs, because it has less built-in image bias (no locality or translation equivariance in the architecture).

I checked the patch arithmetic by hand, then ran Google’s vit-base-patch16-224 (pretrained on ImageNet-21k, fine-tuned on ImageNet-1k) on the public-domain NASA astronaut photo from scikit-image.

import torch
from skimage import data
from transformers import ViTImageProcessor, ViTForImageClassification

img = torch.zeros(1, 3, 224, 224)
patches = img.unfold(2, 16, 16).unfold(3, 16, 16)              # (1, 3, 14, 14, 16, 16)
print("patch grid:", tuple(patches.shape[2:4]), "| tokens:", 14 * 14, "| values per patch:", 3 * 16 * 16)

name = "google/vit-base-patch16-224"
proc = ViTImageProcessor.from_pretrained(name)
vit = ViTForImageClassification.from_pretrained(name).eval()
inputs = proc(images=data.astronaut(), return_tensors="pt")
with torch.no_grad():
    out = vit(**inputs, output_hidden_states=True)
print("sequence length inside ViT:", out.hidden_states[-1].shape[1], "(196 patches + [CLS])")
probs = out.logits.softmax(-1)[0]
for p, i in zip(*probs.topk(3)):
    print(f"{vit.config.id2label[i.item()]:>30}  {p.item():.3f}")

Output:

patch grid: (14, 14) | tokens: 196 | values per patch: 768
sequence length inside ViT: 197 (196 patches + [CLS])
                   oxygen mask  0.170
                 space shuttle  0.148
gasmask, respirator, gas helmet  0.023