Paper summary: The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
BitNet b1.58 restricts every weight to −1, 0 or +1 (about 1.58 bits) and reports matching full-precision LLMs of the same size and data on perplexity and end tasks, at much lower cost.
Problem
This continues BitNet, which I summarised in October 2023: can LLMs be trained with extremely low-precision weights and still match full precision?
Method
BitNet b1.58 makes every weight ternary: −1, 0 or +1. That’s log₂(3) ≈ 1.58 bits. Compared with pure 1-bit weights, the extra 0 lets the model effectively switch features off. Most matrix multiplications become additions and subtractions.
Key results
The abstract claims BitNet b1.58 matches FP16/BF16 Transformers of the same size, trained on the same number of tokens, in both perplexity and end-task performance, while being significantly cheaper in latency, memory, throughput and energy. The authors argue it defines a new scaling law and training recipe, and opens the door to hardware designed for 1-bit LLMs.
Limitations
This is a short paper without a dedicated limitations section. The efficiency benefits depend on hardware and kernels that can exploit ternary weights.
Sources
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (arXiv:2402.17764) — arXiv
- Hugging Face Daily Papers, 2024-02-28 — Hugging Face
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (arXiv HTML full text) — arXiv
This is a summary written from the linked public sources. Figures and numbers are as reported there.