What does it mean for future hardware if it's not using floating point matrix multiplication units?
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
11–20 of 462 posts
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#12Refreshing paper in terms of machine learning papers, simple explanation, easy to replicate, no alchemy-tier interpretations. Can't wait to see this paper replicated or disproved when it comes to real-life production tasks.
The presentation is simplified because it implies knowledge of its predeccesor, BitNet https://arxiv.org/abs/2310.11453
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#13What does it mean for future hardware if it's not using floating point matrix multiplication units?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#14Major breakthrough in LLM scene. Achieve performance and perplexity equivalent to full FP16 models of same parameter size. And you can fit 120B model with a single card 24GB VRAM. This is mind blowing.
I mean, it expands the hardware selection, but until there's models and leader boards etc, can't really say it's a break through.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#15Balanced ternary, my beloved.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#16Refreshing paper in terms of machine learning papers, simple explanation, easy to replicate, no alchemy-tier interpretations. Can't wait to see this paper replicated or disproved when it comes to real-life production tasks.
The presentation is simplified because it implies knowledge of its predeccesor, BitNet https://arxiv.org/abs/2310.11453
Makes sense!
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#17Earlier quoted context omitted.
However they trained their models from scratch, which is also why they only have meaningful numbers for 700M, 1.3B, 3B and 3.9B models. Apparently they are following BitNet's approach of replacing linear layers with quantized layers during training? If it was trivial to convert existing models without performance loss I would have expected them to include a benchmark of that somewhere in the paper to generate even mo…
They present numbers for 7B to 70B models as well.
They do not have perplexity numbers for the larger models (see Table 2), only speed and memory benchmarks.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#18This is great, my employer just gave me a M1 laptop with only 16gb ram and I had to downgrade my 7B parameter local LLM’s to 3 bit quantizing, they’ve been surprisingly okay!
In my personal machine at 64gb ram, I usually use 8x7B at Q5 or 70B at Q4
Its Mistral all the way down! Imagining Q1.58 that’s doing well makes me happy
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#19Earlier quoted context omitted.
However they trained their models from scratch, which is also why they only have meaningful numbers for 700M, 1.3B, 3B and 3.9B models. Apparently they are following BitNet's approach of replacing linear layers with quantized layers during training? If it was trivial to convert existing models without performance loss I would have expected them to include a benchmark of that somewhere in the paper to generate even mo…
They present numbers for 7B to 70B models as well.
Those numbers are for cost only, not performance. It’s not clear they actually trained a 70B vs. just using randomly initialized parameters.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#20Refreshing paper in terms of machine learning papers, simple explanation, easy to replicate, no alchemy-tier interpretations. Can't wait to see this paper replicated or disproved when it comes to real-life production tasks.
The most glaring omission is that they only compared to fp16 models, not to quantized models. And of course the benchmarks might be misleading compared to the real experience.
But if you wanted to make LLM-specific hardware (or x64 instructions tuned for LLMs) this model architecture makes that extremely cheap. Multiplication requires a lot of transistors, this architecture requires only two-bit adders. You could make SIMD instructions that do thousands of these in parallel, for fairly little silicon cost.