Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

271–280 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#271

Post by Noe2097 »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

There is another _shocking_ realization in this work: there are 11 types of people: those who know what binary means, those who don't, and those who say they do but actually don't.

"The era of 1-bit LLMs"

Representing { -1, 0, 1 } can't be done with 1-bit, I'm sorry -- and sad, please let's all get back to something vaguely sound and rigorous.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#272

Post by Klipper3 »

rafaelero wrote: post #101
btbuildem wrote: post #79

Earlier quoted context omitted.

If this dethrones Nvidia, it would be a wonderful side effect

It's more likely that Nvidia will offer support to INT2 in the next generation and keep their dominance.

INT2 ternary is equivalent to INT1 + binary mask. Nvidia supprted INT1 matrix multiply in RTX20 and RTX30 generations, nobody used it, so they removed INT1 support from RTX40 generation.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#273

Post by npunt »

Noe2097 wrote: post #271
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

There is another _shocking_ realization in this work: there are 11 types of people: those who know what binary means, those who don't, and those who say they do but actually don't. "The era of 1-bit LLMs" Representing { -1, 0, 1 } can't be done with 1-bit, I'm sorry -- and sad, please let's all get back to something vaguely sound and rigorous.

Ternary supporters are always bitter about this

(I'll let myself out)

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#274

Post by AaronFriel »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

In undergrad, some of us math majors would joke that there's really only three quantities: 0, 1, infinity.

So, do we need the -1, and/or would a 2.32 bit (5 state, or 6 with +/-0) LLM perform better than a 1.58 bit LLM?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#275

Post by dns_snek »

vidarh wrote: post #134
dns_snek wrote: post #120

Earlier quoted context omitted.

The claim was that "LLMs can do math". Below they linked a model from Google that might be capable of that, but as a general rule (and with OpenAI's models specifically) LLMs can't "do math" by any reasonable definition.

I've had it do plenty of math. Some it does badly at, some it does fine. Generally it's not "disciplined" enough to do things that requires lots of rote repetitive tasks, but neither are most humans, and that has improved drastically as they've adjusted it to instead do what most humans do and use tools. Would it be nice if it also got more willing to "stick to it" when given rote tasks? Sure. But whether or not it c…

It often fails at basic 3-4 digit arithmetic. If you're stretching that definition far enough to claim that GPT4 can "do math" then I should be able to call myself a commercial pilot because I can land a plane in a sim 20% of the time.

I'm not moving goalposts, the original claim was that LLMs can "do math". Primary school arithmetic is math.

GPT-4 can't do math and that's okay, I don't understand why so many of you are so touchy and defensive about this. It's a limitation that exists, nothing more, nothing less.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#276

Post by para_parolu »

pennomi wrote: post #270
SushiHippie wrote: post #189

Earlier quoted context omitted.

But if I understand it correctly, they already need to use 2 bits, one for the sign and another one for the value, so there is already one wasted state, which could be used for -0.

You can pack two trits into three bits, however. So one byte could hold 5 values instead of 4.

Can processor perform addition on them effectively?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#277

Post by fzliu »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

This will be big for FPGAs - adders are extremely cheap compared to multipliers and other DSP blocks.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#278

Post by kouru225 »

SushiHippie wrote: post #253
kouru225 wrote: post #247

Ok can someone catch me up to speed on LLM hardware requirements? Last I looked I needed a 20 gb vram card to run a good one. Is that not true anymore?

Not true anymore, but it also highly depends on what your definition of "a good one" is. Many people find Mistral 7B to be excellent, around gpt-3.5 level of good. Mistral 7B normally requires like 20gb VRAM, but with llama.cpp and quantization, you could even run it on your phone (albeit bad quality). Quantization >= q4_K_M seem to provide nearly as good responses as the unquantized model, and q4_K_M only needs ~7GB…

Oh Jesus so basically it’s very feasible for me to run my own local llm on a NAS or a server or something… well I guess it’s time for me to get on with the times…

Thanks!

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#279

Post by eigenvalue »

Is it really so surprising that something like this works given how human brain neurons work? My admittedly basic understanding is that these operate through an all-or-nothing principle for their action potentials (firing): they either fire or they don't, based on whether the input signals reach a certain threshold. So the output is already sort of binary in biological neurons. The inputs are more like continuous values, since they are the sum of many different neurons sending signals into each neuron, but in this paper the activations are 8-bit, not binary/ternary. Can any neuroscientists here comment?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#280

Post by [deleted] »

fzliu wrote: post #277
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

This will be big for FPGAs - adders are extremely cheap compared to multipliers and other DSP blocks.

[deleted]
Post reply on HN