Paper summary: 2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

Problem

Interleaved image–text corpora help VLMs learn more naturally than image–caption pairs. But web-crawled ones suffer from low knowledge density, loose image–text relations and poor coherence between images. Meanwhile, huge numbers of instructional videos (e.g. online geometry courses) are barely used for training.

Method

The authors build a multimodal textbook:

1. Use an LLM-proposed taxonomy to gather instructional videos systematically: over 2.5 years of video, 22,000 class hours.
2. Progressively extract and refine visual (keyframes), audio (ASR transcripts) and text (OCR) knowledge.
3. Arrange these in time order into an interleaved image–text corpus.

Key results

From the abstract: compared with alternatives, the corpus offers more coherent context, richer knowledge and better image–text alignment. Pre-training on it gives strong results especially on knowledge- and reasoning-heavy tasks such as ScienceQA and MathVista, and better use of interleaved few-shot context.

Limitations

From the paper: despite multi-level filtering, the corpus may still include redundant keyframes and low-quality text. It targets understanding and text generation, since loss isn’t computed on image tokens, so generation models aren’t covered yet.