Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

431–440 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#431

Post by imtringued »

vidarh wrote: post #184
imtringued wrote: post #147

Earlier quoted context omitted.

You do realize that arithmetic is a very simple symbolic manipulation task? All you have to do is keep track of the carry. I haven't seen an LLM that couldn't get digit by digit addition done, but they always mess up the carry.

Just like humans. Try to get regular people do e.g. add 15-16 digit numbers (where is typically where I'd see GPT4 start to get "sloppy" unless you prompt it the way you would a child who's learning and is still prone to get annoyed and wonder why the hell you make them to it manually), and see how many start making mistakes. I find it really comical that this is what people complain about GPT over - there's zero ben…

Is this rant really necessary? Most models, especially ChatGPT4 can perform carry based addition and there is zero reason for them to fail at it, but the moment you start using quantized models such as the 5 bit mixtral 8x7b the quality drops annoyingly. Is it really too much to ask? It's possible and it has been done. Now I'm supposed to whip out a python interpreter for this stuff, because the LLM is literally pretending to be a stupid human, really?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#432

Post by jsnelgro »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

It all seems too good to be true but your comment helped me develop a mental model for how this could work.

The most inspiring aspect to me here is just realizing how much potential low-hanging fruit there is in this space! What other seemingly naïve optimizations are there to try out?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#434

Post by concrete_head »

singularity2001 wrote: post #364
concrete_head wrote: post #197

Earlier quoted context omitted.

It's not quiet spikes but getting closer to the idea. I'm amazed it has taken this long for this type of thing to reach HN which gives next to no attention to spiking neural networks. Simon Thorpe, a CNRS researcher has got some fascinating papers and lectures on YouTube on using binary weights on neuromorphic hardware which has had practical applications for over 20 years already. I made an account just to drop his…

why is his name so dangerous you can't drop it on your main account lel?

It is not. Perhaps I should have clarified that I don't have another account. I've been a lurker until now.

In my time lurking I've noticed that the community here basically focuses solely on the von Neumann architecture. If anyone is interested in delving into the world of spikes he has some interesting ideas and good material available.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#435

Post by rapatel0 »

az226 wrote: post #389
rapatel0 wrote: post #165

The mathematics of the BNNs are sound. The shannon entropy of a word is really small (I vaguely remember ~2 bits). Also all neural networks are ridiculously over provisioned. I worked on 7 years ago trying to efficiently binarize CNNs from existing models. It the difficult was getting training running without the losses going to high. I think that vision models will be much more difficult to binarize, but you might n…

What about text to speech models? Do you think ternary will work?

Just to be clear, it's all theoretically possible. There are already versions of BNN versions of YoLo and other CNNs. No reason why transformers wouldn't work for that or audio. It just might be harder to get them to train well enough.

Speech to text, however, is super interesting. You just gave me an idea! I'm gonna go run some experiments :D

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#436

Post by cs702 »

p1esk wrote: post #264
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Ternary networks have been used since 2015. There are hundreds of papers. They all require full QAT (training from scratch). Not sure why you’re shocked.

Because it's not just the use ternary values. It's also that there are no dot-products; there are only additions. And when we apply both changes to existing LLMs, there's no performance degradation (as tested by the authors).

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#437

Post by Manabu-eo »

SushiHippie wrote: post #189
creshal wrote: post #124

Earlier quoted context omitted.

> Could the additional -0 carry some pseudo-gradient information, ("the 0 leaning towards the negative side")? Probably, but is it worth the cost? One of the goals behind BitNet and this paper is to find a way to implement LLMs as efficiently in hardware as possible, and foregoing floating point semantics is a big part of it. I'm not sure if there's a way to encode -0 that doesn't throw out half the performance gains…

But if I understand it correctly, they already need to use 2 bits, one for the sign and another one for the value, so there is already one wasted state, which could be used for -0.

You can use a bit for zero or non-zero and then use bits only for providing the sign to non-zero values, for example. The sign part will be variable length but can probably be made very fast with hardware support.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#438

Post by craq »

gliptic wrote: post #233
cs702 wrote: post #230

Earlier quoted context omitted.

Thank you. Your key point -- that so far all models with the proposed methods may have been only "grossly trained" -- is compelling. If I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-apples. That seems sensible to me, and makes replication easier, but I agree we need more to see extensive testing, after more extensi…

They also trained 3B with 2 trillion tokens. > The number of training tokens is a crucial factor for LLMs. To test the scalability of BitNet b1.58 in terms of tokens, we trained a BitNet b1.58 model with 2T tokens following the data recipe of StableLM-3B [ TBMR], which is the state-of-the-art open-source 3B model. > [..] > Our findings shows that BitNet b1.58 achieves a superior performance on all end tasks, indicati…

And I was hoping to agree on this, but there is no 'SOTA StableLM-3b' with 2T tokens. Which is a big gap in the paper, because StableLM 3B is trained on 1T tokens for 4 epochs. And the benchmarks they report far exceed the benchmarks shown in the paper. You can find them in the official StableLM git and compare to the results in the paper https://github.com/Stability-AI/StableLM?tab=readme-ov-file#...

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#439

Post by craq »

cs702 wrote: post #135
rhaps0dy wrote: post #132

Earlier quoted context omitted.

I think you need more evidence than this paper (which is very short and light on actual numbers) to be this shocked. For example, most of the plots in the paper are actually of throughput, memory, etc. all performance characteristics that are better on the ternary version. Which, of course. The only thing that contains perplexities are Table 1 and 2. There, they compare "BitNet b1.58 to our reproduced FP16 LLaMA LLM…

Thank you. I think the paper as it is provides enough evidence to support the claims. If I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-apples. That's sensible. It allows for easier replication of the results. Otherwise, I agree with you that more extensive testing, after more extensive pretraining, is still necessa…

And that's true, but why do they limit it to 100B tokens? And why not provide the loss curves in the end to show that both models have converged? What's not proven to me, in this paper, is the ability of the model to scale and generalize to bigger datasets. It's easy to see how a model of sufficient size can overcome the quantization bottleneck, when trained on such a small dataset. Which is perhaps why smaller variations failed.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#440

Post by rep_lodsb »

phkahler wrote: post #412
cs702 wrote: post #406

Earlier quoted context omitted.

Yes. More generally, this will enable implementation via crazy-cheap bit-wise ops in binary hardware, and possibly, maybe, via crazy-cheap trit-wise ops in ternary hardware that manipulates ternary digits, or trits. Note that any binary op over trits has only nine possible (trit, trit) input pairs and only three possible trit outputs. Maybe ternary hardware for AI will become a thing?

Fleshing out my thought above. If we want to multiply A*B = C and all operands are stored in 2 separate bits Ap and An (Ap = 1 if A = +1 while An = 1 if A = -1). We can do a product with: Cp = (Ap & Bp) | (An & Bn) Cn = (An & Bp) | (Ap & Bn) So 64 products in 6 instructions, or 256 in 6 instructions with AVX2, or 512 in six instructions using AVX512. If you can execute 2 instructions at a time on different words, thi…

Another way would be to use one register for "zero" vs. "non-zero", and another for negative (basically 2 bit sign-magnitude representation).

    C_sgn = A_sgn ^ B_sgn
    C_mag = A_mag & B_mag
The result can then be converted into bitmasks for positive and negative:

    C_plus = C_mag & ~C_sgn
    C_minus = C_mag & C_sgn
This solution should be more efficient if there is an "AND NOT" instruction, or when multiplying more than two factors.
Post reply on HN