Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

31–40 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#31

Post by KeplerBoy »

gpderetta wrote: post #24
cyanydeez wrote: post #13

Earlier quoted context omitted.

https://stackoverflow.com/questions/45373679/why-is-it-faste...

As per answer, the reason float is faster than in is because a) hardware companies provide float ALUs than integer ALUs and b) float FMA is a thing, while integer FMA isn't. Both are because currently most HPC-like loads use floats instead of integers, not because of intrinsic hardware reasons.

If it's desired integer performance could far exceed float performance, since ALUs need less die area than FPUs.

If this paper holds, I'd expect that's where custom accelerators will be heading.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#32

Post by IanCal »

leroman wrote: post #25

Can someone versed in the ways of math explain how this is different from previous quantization methods? And specifically, seeing how going from 16fp to 8bit mostly gives same perplexity while anything further seems to lose quality / dumb down the model, how is this even less precise method is able to achieve this?

It's not quantising existing models, they're training new ones.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#33

Post by leroman »

sebzim4500 wrote: post #27
raghavtoshniwal wrote: post #22

Sooo, short Nvidia?

These still run on GPUs

- we have llama.cpp (could be enough or at least as mentioned in the paper a co-processor to accelerate the calc can be added, less need for large RAM / high end hardware)

- as most work is inference, might not need for as many GPUs

- consumer cards (24G) could possibly run the big models

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#35

Post by leroman »

IanCal wrote: post #32
leroman wrote: post #25

Can someone versed in the ways of math explain how this is different from previous quantization methods? And specifically, seeing how going from 16fp to 8bit mostly gives same perplexity while anything further seems to lose quality / dumb down the model, how is this even less precise method is able to achieve this?

It's not quantising existing models, they're training new ones.

I understand this part but it seemed that the 16->8->4 etc is similar to compression of the "net" and seemed to lower quality below 8.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#36

Post by wongarsu »

lucubratory wrote: post #21

After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I do…

I wouldn't be surprised if this causes hardware startups to pop up that build accelerator cards tuned for this architecture. It seems stupidly simple to do inference in hardware, and with most of the training being quantized as well you might even be able to provide speedups (and energy savings) for training with reasonable investment and on cheaper processor nodes than what Nvidia is using.

Sure, Nvidia might eat their lunch in a couple of years, but bitcoin ASICs prove that you can have a niche producing specialized processors, and VCs would probably jump at the thought of disrupting Nvidia's high margin business.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#37

Post by rvnx »

aurareturn wrote: post #29
lucubratory wrote: post #21

After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I do…

After playing with OpenAI's GPT4 API, I'm quite convinced that LLMs would be in everything and everywhere today if inference cost is as low as loading a website and context size is 100x higher. In other words, only inference cost is holding it back from completely changing everything. So if we have a shortcut to getting something like GPT4 to run locally on a small device, watch out.

It's coming in October with the new Apple chip

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#38

Post by sp332 »

1-bit LLMs remind me of a random forum post I read about SACD and limitations of the 1-bit DSD audio format. https://www.audiosciencereview.com/forum/index.php?threads/d... Accumulating approximate values in one bit leads to being "constantly overloaded", with any error correction overwriting all of your real signal from the next step. I think this trinary system might leave enough room to avoid this problem.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#39

Post by llm_trw »

anon373839 wrote: post #28
lucubratory wrote: post #21

After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I do…

It also means the largest models can be scaled up significantly with the same inference budget.

Depends. The only paper they cite for training: https://arxiv.org/pdf/2310.11453.pdf doesn't improve training costs much and most models are already training constrained. Not everyone has $200m to throw at training another model from scratch.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#40

Post by FrustratedMonky »

bmacho wrote: post #30
transfire wrote: post #2

Shouldn’t that be “1-trit”?

Read the pdf https://arxiv.org/pdf/2402.17764.pdf they call it 1-bit everywhere. I don't know why do they do this, 1-bit seems to be a very wrong name for {-1, 0, 1}.

Yes Technically, but it is catchy for the masses. 1-bit seems to get the idea across, even if not technically describing {-1,0,1}.
Post reply on HN