Study
LLMs · Scaling laws
Loss falls predictably as a power law in model size, data and compute; using the Chinchilla paper's fitted formula to see the compute-optimal trade-off between parameters and tokens.
- Scaling laws say loss improves as a smooth power law when you scale parameters, data and compute together.
- Chinchilla's finding: for a fixed compute budget, parameters and training tokens should grow at roughly the same rate.