#scaling

• 7 Jan 2026 · Study

  LLMs · Scaling laws

  Loss falls predictably as a power law in model size, data and compute; using the Chinchilla paper's fitted formula to see the compute-optimal trade-off between parameters and tokens.

• 9 Dec 2024 · Paper summary

  Paper summary: Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

  InternVL 2.5 keeps InternVL 2.0's architecture but improves training, testing and data, and studies scaling of vision encoders, LLMs, data and test-time settings; it's reported to rival GPT-4o and Claude-3.5-Sonnet.

• 27 Dec 2023 · Paper summary

  Paper summary: SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling

  SOLAR 10.7B grows a smaller LLM by duplicating its layer stack, trimming the overlap and continuing pre-training ("depth up-scaling"), a simpler route than mixture-of-experts.