Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

141–150 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#141

Post by jxy »

cs702 wrote: post #128
creshal wrote: post #118

Earlier quoted context omitted.

> * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. Easier said than done, of course, and very impressive that they pulled it off. > In matr…

> Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. I find it shocking that we don't even need lower floating-point precision. We don't need precision at all . We only need three symbols to represent every value. > I feel like this follows naturally from having only ternary values,…

The matrices (weights) are ternary.

The vectors are not.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#142

Post by [deleted] »

cs702 wrote: post #128
creshal wrote: post #118

Earlier quoted context omitted.

> * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. Easier said than done, of course, and very impressive that they pulled it off. > In matr…

> Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. I find it shocking that we don't even need lower floating-point precision. We don't need precision at all . We only need three symbols to represent every value. > I feel like this follows naturally from having only ternary values,…

[deleted]

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#143

Post by cs702 »

jxy wrote: post #141
cs702 wrote: post #128

Earlier quoted context omitted.

> Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. I find it shocking that we don't even need lower floating-point precision. We don't need precision at all . We only need three symbols to represent every value. > I feel like this follows naturally from having only ternary values,…

The matrices (weights) are ternary. The vectors are not.

The activations are in (-1, 1), so they're also representable by (-1, 0, 1).

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#144

Post by BenoitEssiambre »

Low bit parameters is always talked about in terms of performance benefits but I wonder if allowing the LLM to combine parameters to represent values, means it can select the resolution of each value, that is use a kind of internal scientific notation to track the uncertainty of values. More low bit parameters combined together means more precision and resolution, less can mean more uncertainty. This might allow the LLM to better calibrate the uncertainty of it's knowledge in a Bayesian way, to prevent hallucinations from the overconfidence you get from overfitting on too many bits.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#145

Post by imtringued »

creshal wrote: post #138
ynniv wrote: post #82

Earlier quoted context omitted.

You don't need "GPT4" though. Mixtral 8x7B is robust and can be run in 36 Gb, 24 Gb if you're willing to compromise. A 1.5 bit quantization should bring it down to 16. That's still a lot compared to the iPhone 15's 6, but it's close enough to imagine it happening soon. With some kind of streaming-from-flash architecture you might be in the realm already.

> With some kind of streaming-from-flash architecture you might be in the realm already. I thought mmap'ing models to only keep the currently needed pieces in RAM was something that was figured out ~6 months ago? Performance wasn't terribly great iirc, but with how much faster 1.58B is, it should still be okay-ish.

I'm not sure what use that is, other than to maintain the KV cache across requests.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#146

Post by HPsquared »

satellite2 wrote: post #130
creshal wrote: post #118

Earlier quoted context omitted.

> * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. Easier said than done, of course, and very impressive that they pulled it off. > In matr…

Because it's no longer a linear optimization or curve fitting problem. It becomes a voting or combinatorial problem. Which at least in my mind are two completely different areas of research.

With enough parameters, it probably starts looking continuous again. Like how in physics everything is quantised at the smallest scale but if you put enough atoms together it all smooths out and behaves "classically".

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#147

Post by imtringued »

vidarh wrote: post #117
dns_snek wrote: post #108

Earlier quoted context omitted.

Last time I checked, GPT-4 couldn't reliably add 2 numbers, never mind anything more complex.

Last I checked (and confirmed by repeating it just now) GPT-4 did just fine at adding 2 numbers up, because it knows better now than to do that manually and will express it as Python. It does worse if you try to force it to do it step by step like a child and don't reinforce adherence to the rules every step, because just like humans it gets "sloppy" when you try to get it to repeat the same steps over and over. If y…

You do realize that arithmetic is a very simple symbolic manipulation task? All you have to do is keep track of the carry. I haven't seen an LLM that couldn't get digit by digit addition done, but they always mess up the carry.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#148

Post by lr1970 »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Authors reported perplexity only for small up to 3B weights models. On the other hand, they reported throughput for 70B model, but not its performance (perplexity, end-to-end tasks). Very unfortunate omission. Overall, the paper is rather poorly written.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#149

Post by monocasa »

zcw100 wrote: post #104
sigmoid10 wrote: post #56

Earlier quoted context omitted.

People are working on that [1]. In some sense, it's a step back to analog computing. Add/multiply is possible to do directly in memory with voltages, but it's less versatile (and stable) than digital computing. So you can't do all calculations in a neural network that way, meaning some digital components will always be necessary. But I'm pretty sure analog will make a comeback for AI chips sooner or later. [1] https:…

Reminds me of my father saying something about how vacuum tubes are great integrators.

Chips are too. Opamps can add, multiply, subtract, divide, integrate and differentiate depending on how they're plugged in.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#150

Post by cs702 »

lr1970 wrote: post #148
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Authors reported perplexity only for small up to 3B weights models. On the other hand, they reported throughput for 70B model, but not its performance (perplexity, end-to-end tasks). Very unfortunate omission. Overall, the paper is rather poorly written.

If I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-apples. That's sensible. It allows for easier replication of the results. Otherwise, I agree with you that more extensive testing, after more extensive pretraining, at larger model sizes, is still necessary.
Post reply on HN