Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

61–70 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#61

Post by FrustratedMonky »

accurrent wrote: post #6
anon373839 wrote: post #5

> BitNet b1.58 can match the performance of the full precision baseline starting from a 3B size. ... This demonstrates that BitNet b1.58 is a Pareto improvement over the state-of-the-art LLM models. > BitNet b1.58 is enabling a new scaling law with respect to model performance and inference cost. As a reference, we can have the following equivalence between different model sizes in 1.58-bit and 16-bit based on the re…

They seem to be using LLAMA. Might be worth trying out. Their conversion formula seems stupidly simple.

Yes. I wonder then how long before someone that does have a lot of compute power like OpenAI/MS, or others, can rapidly pivot and try this out on some even larger models.

Doesn't this mean that current big players can rapidly expand by huge multiples in size.?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#62

Post by wakawaka28 »

osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

I have heard of people trying to build analog AI devices but that seems like years ago, and no news has come out about it in recent times. Maybe it is harder than it seems. I bet it is expensive to regulate voltage so precisely and it's not a flexible enough scheme to be support training neural networks like we have now, which are highly reconfigurable. I've also heard of people trying to use analog computing for more mundane things. But no devices have hit the market after so many years so I'm assuming it is a super hard problem, maybe even intractible.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#63

Post by loudmax »

osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

The Veritasium Youtube channel did a video about this about a year ago: https://www.youtube.com/watch?v=GVsUOuSjvcg

They visit Texas company Mythic AI to discuss how they use flash memory for machine learning. There's a California company named Syntiant doing something similar.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#64

Post by declaredapple »

aurareturn wrote: post #29
lucubratory wrote: post #21

After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I do…

After playing with OpenAI's GPT4 API, I'm quite convinced that LLMs would be in everything and everywhere today if inference cost is as low as loading a website and context size is 100x higher. In other words, only inference cost is holding it back from completely changing everything. So if we have a shortcut to getting something like GPT4 to run locally on a small device, watch out.

I'll agree with you, and add that inference speed is a big factor too.

SDXL-ligtning/cascade can generate images in 200ms which is fast enough to fit in a web request, and paradoxically makes it even cheaper to generate.

And using groq at 500 t/s is wild compared to any of the other platforms.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#65

Post by ignoramous »

anon373839 wrote: post #5

> BitNet b1.58 can match the performance of the full precision baseline starting from a 3B size. ... This demonstrates that BitNet b1.58 is a Pareto improvement over the state-of-the-art LLM models. > BitNet b1.58 is enabling a new scaling law with respect to model performance and inference cost. As a reference, we can have the following equivalence between different model sizes in 1.58-bit and 16-bit based on the re…

I wonder if 1bit quantization is the main reason why pplx.ai is faster than any other RAG or chatbot. For instance, Gemini in comparison is a turtle, though it is better at explanations, while pplx is concise.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#66

Post by baq »

londons_explore wrote: post #47

Powers of 3 don't pack well into binary memory... A 1 bit multiplier in silicon is a single logic gate, but a ternary decoder to decode a packed tri-state 'weight' is bigger. I therefore suspect that this method will be extended to make all weights simple 1 or 0 (ie. Binary). Perhaps that will be done by having half the weights have 1 or 0 values, while the other half are -1 or 0.

You can build dedicated silicon with ternary gates: https://medium.com/@rxseger/exploring-ternary-logic-tnand-an...

Not sure if it's more efficient than just binary digital circuits in highly integrated chip, though.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#67

Post by imjonse »

sp332 wrote: post #17
imjonse wrote: post #10

Earlier quoted context omitted.

They present numbers for 7B to 70B models as well.

They do not have perplexity numbers for the larger models (see Table 2), only speed and memory benchmarks.

You're both right, I skimmed the paper, saw large model numbers but didn't notice it was for speed. On the HF page they say those models are being trained.

https://huggingface.co/papers/2402.17764

"We haven't finished the training of the models beyond 3B as it requires much much more resources. However, we're optimistic about the results because we have verified that BitNet follows a similar performance-parameter scaling law as the full-precision LLMs. We'll update the results on larger models once they're ready."

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#68

Post by btbuildem »

anon373839 wrote: post #5

> BitNet b1.58 can match the performance of the full precision baseline starting from a 3B size. ... This demonstrates that BitNet b1.58 is a Pareto improvement over the state-of-the-art LLM models. > BitNet b1.58 is enabling a new scaling law with respect to model performance and inference cost. As a reference, we can have the following equivalence between different model sizes in 1.58-bit and 16-bit based on the re…

Discussion on HF [1] implies that no, conversion is not helpful. It would take training the model from scratch.

1: https://huggingface.co/papers/2402.17764

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#69

Post by jmmcd »

QuesnayJr wrote: post #3
transfire wrote: post #2

Shouldn’t that be “1-trit”?

They call it 1.58-bit in the paper. (1.58 is roughly the base 2 logarithm of 3.)

So by “1-bit” they mean “less than 2 bits”. AI is an insufferable field at times like this.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#70

Post by rfoo »

KeplerBoy wrote: post #26
the8472 wrote: post #11

What does it mean for future hardware if it's not using floating point matrix multiplication units?

Expect Nvidia to advertise with their TOPS numbers instead of their FLOPS.

Already happened years ago. They advertised TOPS for int8/int4 [0], and with 50% sparsity [1].

[0] low-bit CNNs worked pretty well actually.

[1] Totally useless marketing snake oil.

Post reply on HN