Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

101–110 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#101

Post by rafaelero »

btbuildem wrote: post #79
lucubratory wrote: post #21

After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I do…

If this dethrones Nvidia, it would be a wonderful side effect

It's more likely that Nvidia will offer support to INT2 in the next generation and keep their dominance.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#102

Post by FergusArgyll »

yieldcrv wrote: post #18

This is great, my employer just gave me a M1 laptop with only 16gb ram and I had to downgrade my 7B parameter local LLM’s to 3 bit quantizing, they’ve been surprisingly okay! In my personal machine at 64gb ram, I usually use 8x7B at Q5 or 70B at Q4 Its Mistral all the way down! Imagining Q1.58 that’s doing well makes me happy

You shouldn't have to quantize it that much, maybe you're running a lot of other programs while running inference?

Also, try using pure llama.cpp, AFAIK it's the least possible overhead

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#103

Post by jart »

rafaelero wrote: post #99
jart wrote: post #60

Earlier quoted context omitted.

LLMs will give normal people a firmer standing in technological society. That's a good thing. But will it change everything? Not a chance. Even if LLMs did change everything, that probably would not be a good thing. Dijkstra says Muslim algebra died when it returned to the rhetoric style, and the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters…

LLM's can do math as well.

What makes you think that? Which LLMs?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#104

Post by zcw100 »

sigmoid10 wrote: post #56
osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

People are working on that [1]. In some sense, it's a step back to analog computing. Add/multiply is possible to do directly in memory with voltages, but it's less versatile (and stable) than digital computing. So you can't do all calculations in a neural network that way, meaning some digital components will always be necessary. But I'm pretty sure analog will make a comeback for AI chips sooner or later. [1] https:…

Reminds me of my father saying something about how vacuum tubes are great integrators.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#105

Post by londons_explore »

tromp wrote: post #73
londons_explore wrote: post #47

Powers of 3 don't pack well into binary memory... A 1 bit multiplier in silicon is a single logic gate, but a ternary decoder to decode a packed tri-state 'weight' is bigger. I therefore suspect that this method will be extended to make all weights simple 1 or 0 (ie. Binary). Perhaps that will be done by having half the weights have 1 or 0 values, while the other half are -1 or 0.

5 trits fit into 1 byte pretty well, since 3^5 = 243 is just under 2^8 = 256. That should be called an 8/5 = 1.6 bit model though, while the paper names it 1.58 bit, closer to log_2(3) ~ 1.5849625

But the decoder for that will be 25+ gates, which is huge compared to the handful of gates to use the resulting weights.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#107

Post by riskable »

jmmcd wrote: post #69
QuesnayJr wrote: post #3

Earlier quoted context omitted.

They call it 1.58-bit in the paper. (1.58 is roughly the base 2 logarithm of 3.)

So by “1-bit” they mean “less than 2 bits”. AI is an insufferable field at times like this.

What else are they going to call it? Nobody wants to say they wrote some two-bit paper about AI!

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#108

Post by dns_snek »

rafaelero wrote: post #99
jart wrote: post #60

Earlier quoted context omitted.

LLMs will give normal people a firmer standing in technological society. That's a good thing. But will it change everything? Not a chance. Even if LLMs did change everything, that probably would not be a good thing. Dijkstra says Muslim algebra died when it returned to the rhetoric style, and the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters…

LLM's can do math as well.

Last time I checked, GPT-4 couldn't reliably add 2 numbers, never mind anything more complex.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#109

Post by ekianjo »

rafaelero wrote: post #99
jart wrote: post #60

Earlier quoted context omitted.

LLMs will give normal people a firmer standing in technological society. That's a good thing. But will it change everything? Not a chance. Even if LLMs did change everything, that probably would not be a good thing. Dijkstra says Muslim algebra died when it returned to the rhetoric style, and the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters…

LLM's can do math as well.

most open models do it poorly though. ChatGPT is better at it.
Post reply on HN