Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

91–100 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#91

Post by BlueTemplar »

osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

I have heard that the first commercial neural network chip (by Intel, in the 90s) was analog ?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#94

Post by paipa »

edflsafoiewq wrote: post #46
bmacho wrote: post #30

Earlier quoted context omitted.

Read the pdf https://arxiv.org/pdf/2402.17764.pdf they call it 1-bit everywhere. I don't know why do they do this, 1-bit seems to be a very wrong name for {-1, 0, 1}.

I think 0 "doesn't count", since you don't have to add or subtract anything for it, just mask it out.

Would be cool to see what happens if you quantize towards zero preferentially. Sparsifying the matrix should improve inference speed directly, right?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#95

Post by leroman »

sebzim4500 wrote: post #75
leroman wrote: post #33

Earlier quoted context omitted.

- we have llama.cpp (could be enough or at least as mentioned in the paper a co-processor to accelerate the calc can be added, less need for large RAM / high end hardware) - as most work is inference, might not need for as many GPUs - consumer cards (24G) could possibly run the big models

If consumer cards can run the big models, then datacenter cards will be able to efficiently run the really big models.

Some tasks we are using LLMs for are performing very close to GPT-4 levels using 7B models, so really depends on what value you are looking to get.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#96

Post by barrenko »

osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

Hmm, maybe some (signaling) inspiration from biology other than neural signaling.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#98

Post by woadwarrior01 »

yieldcrv wrote: post #18

This is great, my employer just gave me a M1 laptop with only 16gb ram and I had to downgrade my 7B parameter local LLM’s to 3 bit quantizing, they’ve been surprisingly okay! In my personal machine at 64gb ram, I usually use 8x7B at Q5 or 70B at Q4 Its Mistral all the way down! Imagining Q1.58 that’s doing well makes me happy

You can run 4 bit quantized versions of SOLAR-10.7B and Llama 2 13B based models quite well on 16GB M1 laptops.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#99

Post by rafaelero »

jart wrote: post #60
aurareturn wrote: post #29

Earlier quoted context omitted.

After playing with OpenAI's GPT4 API, I'm quite convinced that LLMs would be in everything and everywhere today if inference cost is as low as loading a website and context size is 100x higher. In other words, only inference cost is holding it back from completely changing everything. So if we have a shortcut to getting something like GPT4 to run locally on a small device, watch out.

LLMs will give normal people a firmer standing in technological society. That's a good thing. But will it change everything? Not a chance. Even if LLMs did change everything, that probably would not be a good thing. Dijkstra says Muslim algebra died when it returned to the rhetoric style, and the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters…

LLM's can do math as well.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#100

Post by cs702 »

There are two findings I find shocking in this work:

* In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1).

* In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the paper for exact details.

On existing hardware, the gains in compute and memory efficiency are significant, without performance degradation (as tested by the authors).

If the proposed methods are implemented in hardware, we will see even greater gains in compute and memory efficiency.

Wow.

Post reply on HN