Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

71–80 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#71

Post by imjonse »

bArray wrote: post #58
imjonse wrote: post #7

Too bad there seem to be no pretrained models to download. This is not a quantization method to apply on existing models, so having the pretrained weights is needed if one wants to test it.

+1 On this, the real proof would have been testing both models side-by-side. It seems that it may be published on GitHub [1] according to HuggingFace [2]. [1] https://github.com/microsoft/unilm/tree/master/bitnet [2] https://huggingface.co/papers/2402.17764

Nothing there yet, but it's good to know they want to publish just did not get around to yet.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#72

Post by jairuhme »

sigmoid10 wrote: post #50
rvnx wrote: post #37

Earlier quoted context omitted.

It's coming in October with the new Apple chip

I'd be very surprised if Apple can put something on the level of GPT4 on a handheld. Remember, GPT4 is estimated to be around 1.7 trillion parameters. That's 3.4TB at 16 bit and it would still be ~340GB at 1.58bits. The best we can hope for is a low-ish level few billion parameter model. Which would still be cool on a phone, but as of today these models are nowhere near GPT4.

They won't have something at that size because as you pointed out, it is still huge. But depending on how they are used, smaller parameter models may be better for specific on-phone tasks that start to make the size of the model not a problem. GPT4 is so large because it is very general purpose with the goal seeming to be to answer anything. You could have a smaller model focused solely on Siri or something that wouldn't require the parameter size of GPT4

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#73

Post by tromp »

londons_explore wrote: post #47

Powers of 3 don't pack well into binary memory... A 1 bit multiplier in silicon is a single logic gate, but a ternary decoder to decode a packed tri-state 'weight' is bigger. I therefore suspect that this method will be extended to make all weights simple 1 or 0 (ie. Binary). Perhaps that will be done by having half the weights have 1 or 0 values, while the other half are -1 or 0.

5 trits fit into 1 byte pretty well, since 3^5 = 243 is just under 2^8 = 256.

That should be called an 8/5 = 1.6 bit model though, while the paper names it 1.58 bit, closer to log_2(3) ~ 1.5849625

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#74

Post by numpad0 »

edflsafoiewq wrote: post #46
bmacho wrote: post #30

Earlier quoted context omitted.

Read the pdf https://arxiv.org/pdf/2402.17764.pdf they call it 1-bit everywhere. I don't know why do they do this, 1-bit seems to be a very wrong name for {-1, 0, 1}.

I think 0 "doesn't count", since you don't have to add or subtract anything for it, just mask it out.

Ternary or three-value logic is a thing in CS[1]

1: https://en.wikipedia.org/wiki/Three-valued_logic

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#75

Post by sebzim4500 »

leroman wrote: post #33
sebzim4500 wrote: post #27

Earlier quoted context omitted.

These still run on GPUs

- we have llama.cpp (could be enough or at least as mentioned in the paper a co-processor to accelerate the calc can be added, less need for large RAM / high end hardware) - as most work is inference, might not need for as many GPUs - consumer cards (24G) could possibly run the big models

If consumer cards can run the big models, then datacenter cards will be able to efficiently run the really big models.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#76

Post by sebzim4500 »

londons_explore wrote: post #52
sebzim4500 wrote: post #27

Earlier quoted context omitted.

These still run on GPUs

GPU's aren't yet awfully efficient at 1 bit math. I could imagine FPGA designs might be competitive. And dedicated ASIC's would almost certainly beat both by a decent margin.

I'm very unconvinced that ASICs are better suited for this than for FP16/FP8 models that are being used today.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#77

Post by gryn »

osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

the reason why digital/numeric processing won is the power loss in the analog world. when design an analog circuit the next processing stage you add at the end has impact on the ones before it.

this then require a higher skill from the engineers/consumers.

if you want to avoid that you need to add op-amps with a gain of 1 at the boundary of each one, this also that care of the power loss at each stage.

the other part is that there's a limit of to the amount of useful information/computation you can do with analog processing too once you take into account voltage noise. when you do a comparison there are stages where analog win but also place where where digital wins.

I'll edit later this with a link to some papers that discuss these topics if I manage to find them in my mess.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#78

Post by joelthelion »

Assuming this is confirmed, what's the impact on training?

Inference is definitely an issue for LLMs right now. But if training were suddenly possible for lone hackers (or maybe smaller companies), it would open up a lot of new possibilities as well.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#79

Post by btbuildem »

lucubratory wrote: post #21

After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I do…

If this dethrones Nvidia, it would be a wonderful side effect

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#80

Post by osigurdson »

wakawaka28 wrote: post #62
osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

I have heard of people trying to build analog AI devices but that seems like years ago, and no news has come out about it in recent times. Maybe it is harder than it seems. I bet it is expensive to regulate voltage so precisely and it's not a flexible enough scheme to be support training neural networks like we have now, which are highly reconfigurable. I've also heard of people trying to use analog computing for mor…

Perhaps another variation on the idea is to allow a higher error rate. For example, if a 0.01% error rate was acceptable in AI, perhaps the voltage range between states could be lowered (which has a quadratic relationship to power consumption) and clock speed could increase.
Post reply on HN