Paper summary: Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Problem

Following phi-1.5 (summarised in September 2023), the question is how capable a model small enough for a phone can be.

Method

phi-3-mini has 3.8 billion parameters and is trained on 3.3 trillion tokens. The data scales up the phi-2 recipe: heavily filtered public web data plus synthetic data. The model is then aligned for robustness, safety and chat format. The report adds:

• phi-3-small (7B) and phi-3-medium (14B), trained on 4.8T tokens
• a phi-3.5 series: phi-3.5-mini, phi-3.5-MoE (16 × 3.8B experts, 6.6B active) and phi-3.5-Vision

Key results

As reported in the abstract:

• phi-3-mini: 69% on MMLU and 8.38 on MT-bench, which the authors compare with Mixtral 8x7B and GPT-3.5
• phi-3-small / phi-3-medium: 75% / 78% MMLU and 8.7 / 8.9 MT-bench

Limitations

The extracted text has no dedicated limitations section. Benchmark comparisons rely partly on “internal testing”, as the abstract says. The report itself notes that phi-3-mini is fundamentally limited by its size for some tasks: it doesn’t have the capacity to store much factual knowledge (visible in its lower TriviaQA score), which the authors suggest can be offset by pairing it with a search engine.