The Era of 1-bit LLMs: ternary parameters for cost-effective computing
1–10 of 462 posts
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#2Shouldn’t that be “1-trit”?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#3Shouldn’t that be “1-trit”?
They call it 1.58-bit in the paper. (1.58 is roughly the base 2 logarithm of 3.)
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#4Major breakthrough in LLM scene. Achieve performance and perplexity equivalent to full FP16 models of same parameter size.
And you can fit 120B model with a single card 24GB VRAM. This is mind blowing.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#5> BitNet b1.58 can match the performance of the full precision baseline starting from a 3B size. ... This demonstrates that BitNet b1.58 is a Pareto improvement over the state-of-the-art LLM models.
> BitNet b1.58 is enabling a new scaling law with respect to model performance and inference cost. As a reference, we can have the following equivalence between different model sizes in 1.58-bit and 16-bit based on the results in Figure 2 and 3.
> • 13B BitNet b1.58 is more efficient, in terms of latency, memory usage and energy consumption, than 3B FP16 LLM.
> • 30B BitNet b1.58 is more efficient, in terms of latency, memory usage and energy consumption, than 7B FP16 LLM.
> • 70B BitNet b1.58 is more efficient, in terms of latency, memory usage and energy consumption, than 13B FP16 LLM.
This paper seems to represent a monumental breakthrough in LLM efficiency, as the efficiency gains come with zero (or negative) performance penalty.
Does it seem at all likely that existing models could be converted?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#6> BitNet b1.58 can match the performance of the full precision baseline starting from a 3B size. ... This demonstrates that BitNet b1.58 is a Pareto improvement over the state-of-the-art LLM models. > BitNet b1.58 is enabling a new scaling law with respect to model performance and inference cost. As a reference, we can have the following equivalence between different model sizes in 1.58-bit and 16-bit based on the re…
They seem to be using LLAMA. Might be worth trying out. Their conversion formula seems stupidly simple.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#7Too bad there seem to be no pretrained models to download. This is not a quantization method to apply on existing models, so having the pretrained weights is needed if one wants to test it.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#8> BitNet b1.58 can match the performance of the full precision baseline starting from a 3B size. ... This demonstrates that BitNet b1.58 is a Pareto improvement over the state-of-the-art LLM models. > BitNet b1.58 is enabling a new scaling law with respect to model performance and inference cost. As a reference, we can have the following equivalence between different model sizes in 1.58-bit and 16-bit based on the re…
They seem to be using LLAMA. Might be worth trying out. Their conversion formula seems stupidly simple.
However they trained their models from scratch, which is also why they only have meaningful numbers for 700M, 1.3B, 3B and 3.9B models. Apparently they are following BitNet's approach of replacing linear layers with quantized layers during training? If it was trivial to convert existing models without performance loss I would have expected them to include a benchmark of that somewhere in the paper to generate even more impact.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#9Refreshing paper in terms of machine learning papers, simple explanation, easy to replicate, no alchemy-tier interpretations. Can't wait to see this paper replicated or disproved when it comes to real-life production tasks.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#10Earlier quoted context omitted.
They seem to be using LLAMA. Might be worth trying out. Their conversion formula seems stupidly simple.
However they trained their models from scratch, which is also why they only have meaningful numbers for 700M, 1.3B, 3B and 3.9B models. Apparently they are following BitNet's approach of replacing linear layers with quantized layers during training? If it was trivial to convert existing models without performance loss I would have expected them to include a benchmark of that somewhere in the paper to generate even mo…
They present numbers for 7B to 70B models as well.