Paper summary: LLaVA-CoT: Let Vision Language Models Reason Step-by-Step
LLaVA-CoT trains a vision-language model to reason in explicit stages (summary, visual interpretation, logical reasoning, conclusion) and adds a stage-wise retracing search at test time.
Problem
VLMs often struggle with systematic, structured reasoning on complex visual questions. Chain-of-thought prompting helps LLMs, but VLMs don’t reliably follow such prompts.
Method
LLaVA-CoT is trained to go through sequential stages on its own:
- summarisation of the question
- visual interpretation (a caption focused on what matters)
- logical reasoning
- conclusion
The training data, LLaVA-CoT-100k, combines samples from various visual question-answering sources with structured reasoning annotations. At test time, a stage-wise retracing search (SWIRES) explores alternatives at the stage level and can go back when a stage looks wrong.
Key results
The abstract reports marked improvements on reasoning-intensive tasks compared with the base model.
Limitations
From the paper: the model sometimes gets lost during retracing or hallucinates to reach an answer, and some input images are too complex for its visual understanding, so even retracing doesn’t help.
Sources
- LLaVA-CoT: Let Vision Language Models Reason Step-by-Step (arXiv:2411.10440) — arXiv
- Hugging Face Daily Papers, 2024-11-17 — Hugging Face
- LLaVA-CoT: Let Vision Language Models Reason Step-by-Step (arXiv HTML full text) — arXiv
- Code repository (linked in the paper) — GitHub
This is a summary written from the linked public sources. Figures and numbers are as reported there.