Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

191–200 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#191

Post by mise_en_place »

vessenes wrote: post #181
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

I'd be VERRY cautious about being excited here. My priors are like this: 1. Initial training of a neural network moves all weights around a large amount at first. 2. Later training of the network adjusts them a small amount. 3. An undertrained network will therefore look a lot like figuring out "positive, negative, or 0?" for each node during early training. If all these things are true, then 1. Early training of an…

Intuitively I've always been a bit skeptical of quantization. Wouldn't there be a tiny loss in precision by doing this type of quantization? I could imagine the error function increasing by utilizing these types of techniques.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#192

Post by SushiHippie »

bArray wrote: post #58
imjonse wrote: post #7

Too bad there seem to be no pretrained models to download. This is not a quantization method to apply on existing models, so having the pretrained weights is needed if one wants to test it.

+1 On this, the real proof would have been testing both models side-by-side. It seems that it may be published on GitHub [1] according to HuggingFace [2]. [1] https://github.com/microsoft/unilm/tree/master/bitnet [2] https://huggingface.co/papers/2402.17764

From [2]:

> We would definitely be happy to open-source the models for future research. Please stay tuned!

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#193

Post by drexlspivey »

osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

Next Up: Quantum AI

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#194

Post by ncruces »

creshal wrote: post #118
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

> * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. Easier said than done, of course, and very impressive that they pulled it off. > In matr…

Well I guess it's the “blowing up parameter count to make up for it” that confuses me, but maybe it's just ignorance.

Like what would be the expected factor of this blow up to make up the difference between ternary and whatever 16 bits encoding they were using?

I mean intuitively I'd expect to need ~10× the symbols to encode the same information? Are they using an order of magnitude more parameters, or is that not how it works?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#195

Post by no_identd »

gojomo wrote: post #177

That's not a 'bit' ("Binary digIT"). It's closer to a 'trit' ("TeRnary-digIT"). Specifically, ternary digits spanning {-1, 0, 1} (rather than the usual {0, 1, 2} in a base-3 numbering system) are 'balanced ternary'. A great intro to the theoretical reasons ternary might have some promise in computing is this 2001 article from 'American Scientist', "Third Base", which quotes Knuth calling balanced-ternary "perhaps the…

See also:

https://en.wikipedia.org/wiki/Nat_(unit) (make sure to read the footnotes, too)

Edit: See also also, on the radix economy of balanced ternary (called "tristate") vs base 3: https://web.archive.org/web/20090312094241/http://abhijit.in... + a wild Marvin Minsky appears: https://archive.fo/gL2Bv

That page also brings up the whole "but division" problem with balanced ternary, however, I personally suspect that http://degiorgi.math.hr/aaa_sem/Div_Krishna/887-889.pdf ("A Division Algorithm for Signed-Digit Arithmetic" by Chin Tung, from 1968 !) might offer an overlooked path to a solution to that problem

And see also also², this quote from TAOCP:

"Cauchy pointed out that negative digits make it unneccesary for a person to memorize the multiplication table past 5x5."

The—INCREDIBLY ANNOYING TO LOCATE—source for which is "105. Calculs numériques. sur les moyens d'éviter les erreurs dans les calculs numériques." on Pdf page 445/document page 431 here:

https://www.e-rara.ch/download/pdf/5702285?name=Tome%2520V%4...

See also also³: https://pdfs.semanticscholar.org/5f77/b1cf105024b41b6824ba91... (Vince, Andrew - Radix Representation and Rep-Tiling)

( +a vaguely related paper here on quantum mechanics & radix economy, BUT it makes the mistake of using an overly specific formula applicable only to unsigned-digit representations thus drawing the wrong conclusions: https://www.researchgate.net/profile/Vladimir_Garcia-Morales... )

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#196

Post by karmasimida »

This is exciting news, if the 8B numbers are true, we can already use model like Mixtral 8x7, even with a single GPU?

But further into the development, we need comparison to large model sizes. 70B might be too much to ask, but 13B should be there at least.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#197

Post by concrete_head »

singularity2001 wrote: post #151

So we almost go back full circle to human (animal) brain binary spikes?

It's not quiet spikes but getting closer to the idea. I'm amazed it has taken this long for this type of thing to reach HN which gives next to no attention to spiking neural networks.

Simon Thorpe, a CNRS researcher has got some fascinating papers and lectures on YouTube on using binary weights on neuromorphic hardware which has had practical applications for over 20 years already.

I made an account just to drop his name somewhere on this forum.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#199

Post by Avisite »

Does quantization need to be an all or nothing? with the kind of low bit models we have seen, my assumption would be that only certain weights would benefit from the extra precision. A mixture of precision with 2-bit, 3-bit, to 8-bit weights might perform well, but I am unsure if any training process could identify the weights that need the extra precision.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#200

Post by rf15 »

m3kw9 wrote: post #182
nutanc wrote: post #122

Earlier quoted context omitted.

We have been experimenting with the paper( https://www.researchgate.net/publication/372834606_ON_NON-IT... ). There is a mathematical proof that binary representation is enough to capture the latent space. And in fact we don't even need to do "training" to get that representation. The practical application we tried out for this algorithm was to create an alternate space for mpnet embeddings of Wikipedia paragraphs. U…

How is this not lossy compression?

It kind of is!
Post reply on HN