Paper summary
Paper summary: Textbooks Are All You Need II: phi-1.5 technical report
phi-1.5 is a 1.3B-parameter model trained mostly on synthetic "textbook-quality" data, reported to match models about five times larger on common-sense reasoning.
- The bet is on data quality over size: synthetic textbook-style data instead of mostly web text.
- The authors report that phi-1.5 also shows familiar LLM failure modes, such as hallucination, but less toxic output.