Paper summary
Paper summary: BitNet: Scaling 1-bit Transformers for Large Language Models
BitNet replaces nn.Linear with BitLinear and trains Transformers with 1-bit weights from scratch, reporting competitive perplexity with much lower memory and energy than FP16 or 8-bit baselines.
- Training with 1-bit weights from the start, rather than quantising afterwards, is the core idea.
- BitNet follows a scaling law similar to full-precision Transformers, according to the authors.