Paper summary: Textbooks Are All You Need II: phi-1.5 technical report

Problem

Can a small language model behave like a much larger one if the training data is good enough? This continues earlier work: TinyStories (a 10M-parameter model producing coherent English) and phi-1 (a 1.3B coding model trained on “textbook quality” data).

Method

phi-1.5 keeps the same small size (1.3 billion parameters) but targets common-sense reasoning in natural language. It’s trained mainly on a synthetic dataset generated by existing LLMs and designed to read like textbooks, largely without web data. The model is open-sourced for research.

Key results

The authors report:

• natural-language performance comparable to models about 5× larger
• better results than most non-frontier LLMs on harder reasoning such as grade-school maths and basic coding
• signs of “thinking step by step” and rudimentary in-context learning

They read this as evidence that data quality can matter more than scale alone.

Limitations

The paper says plainly that phi-1.5 shares weaknesses of larger LLMs, including hallucinations and potential toxic or biased outputs. They do see improvement there, which they attribute to excluding web data. Because the data is synthetic and generated by other LLMs, the dataset itself is hard for outsiders to inspect fully.