Paper summary: LLM360: Towards Fully Transparent Open-Source LLMs

Problem

Many “open” LLMs release only final weights and inference code, and technical reports increasingly stay at a high level. Others then have to rediscover training details, and the field loses insight into how models learn.

Method

LLM360 is an initiative to release the entire training process: pre-training code, configurations and hyperparameters, intermediate checkpoints, optimizer states, the exact data sequence and the data-processing code, plus analyses.

Key results

As a first step, the authors release two 7B models pre-trained from scratch:

• Amber: a general-purpose English LLM
• CrystalCoder: pre-trained specifically for code generation

Everything is published through the LLM360 site, with larger models announced as in progress.

Limitations

The paper is an announcement and a framework rather than a benchmark win. The two first models are 7B, and their quality isn’t the main claim. The value is in the openness of the artifacts.