Paper summary: BitNet: Scaling 1-bit Transformers for Large Language Models

Problem

LLMs are large and energy-hungry to serve. Post-training quantisation to 8 or 4 bits helps, but accuracy drops as you go lower. Could a model be trained with extremely low-precision weights from the start?

Method

BitNet introduces BitLinear, a drop-in replacement for PyTorch’s nn.Linear in which the weights are binarised to 1 bit during the forward pass. Full-precision latent weights are kept for training updates, and activations are quantised too. The rest of the Transformer stays the same, so the architecture can be trained from scratch at scale.

Key results

The authors report that BitNet:

• reaches competitive perplexity and downstream performance
• substantially reduces memory footprint and energy consumption compared with FP16 Transformers and state-of-the-art 8-bit quantisation
• follows a scaling law similar to full-precision Transformers, which suggests it can keep improving with size

Limitations

The paper doesn’t have a dedicated limitations section in the extracted text. Its comparisons are against FP16 Transformers and 8-bit quantisation baselines in the authors’ own experimental setup.