Generative AI · Multimodal models (CLIP)

CLIP (Radford et al., 2021) connects vision and language:

• an image encoder (ViT or ResNet) and a text encoder (Transformer)
• trained on image–caption pairs with a contrastive loss: in a batch of N pairs, the N matching pairs should score high and the N² − N mismatched ones low
• the result is one shared embedding space for images and text

This enables zero-shot classification: write the class names as prompts (“a photo of a …”), embed them, and pick the prompt closest to the image. It’s also how text guides image generation: Stable Diffusion’s text encoder comes from CLIP.

import torch
from transformers import CLIPModel, CLIPProcessor
from skimage import data
from sklearn.datasets import load_sample_image

name = "openai/clip-vit-base-patch32"
model, proc = CLIPModel.from_pretrained(name).eval(), CLIPProcessor.from_pretrained(name)

images = {"astronaut (skimage)": data.astronaut(), "coffee (skimage)": data.coffee(),
          "china.jpg (sklearn)": load_sample_image("china.jpg"), "flower.jpg (sklearn)": load_sample_image("flower.jpg")}
labels = ["an astronaut", "a cup of coffee", "a temple or palace", "a flower", "a cat", "a car"]
prompts = [f"a photo of {l}" for l in labels]

inputs = proc(text=prompts, images=list(images.values()), return_tensors="pt", padding=True)
with torch.no_grad():
    out = model(**inputs)
probs = out.logits_per_image.softmax(-1)
for (name_, _), p in zip(images.items(), probs):
    top = p.topk(2)
    print(f"{name_:>22}: " + ", ".join(f"{labels[i]} {v:.2f}" for v, i in zip(top.values, top.indices)))
print("embedding size:", out.image_embeds.shape[-1], "| image and text share it:", out.image_embeds.shape[-1] == out.text_embeds.shape[-1])

Output:

   astronaut (skimage): an astronaut 1.00, a cat 0.00
      coffee (skimage): a cup of coffee 1.00, a cat 0.00
   china.jpg (sklearn): a temple or palace 1.00, a flower 0.00
  flower.jpg (sklearn): a flower 1.00, a cup of coffee 0.00
embedding size: 512 | image and text share it: True