Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

231–240 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#231

Post by vessenes »

svantana wrote: post #216
vessenes wrote: post #181

Earlier quoted context omitted.

I'd be VERRY cautious about being excited here. My priors are like this: 1. Initial training of a neural network moves all weights around a large amount at first. 2. Later training of the network adjusts them a small amount. 3. An undertrained network will therefore look a lot like figuring out "positive, negative, or 0?" for each node during early training. If all these things are true, then 1. Early training of an…

Wait, are we reading the same paper? What I'm seeing is comparable accuracy to unquantized models for <4B params, and nothing reported for larger models except resource consumption.

Nope, you're right, I got the table inverted in my head. I'm updating my top comment.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#232

Post by vessenes »

vessenes wrote: post #181
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

I'd be VERRY cautious about being excited here. My priors are like this: 1. Initial training of a neural network moves all weights around a large amount at first. 2. Later training of the network adjusts them a small amount. 3. An undertrained network will therefore look a lot like figuring out "positive, negative, or 0?" for each node during early training. If all these things are true, then 1. Early training of an…

Update - I'm still cautious about this paper, but I had the table numbers inverted in my head while thinking about it. The paper shows better perplexity results than competing models at larger parameter sizes, so I was wrong.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#233

Post by gliptic »

cs702 wrote: post #230
vessenes wrote: post #181

Earlier quoted context omitted.

I'd be VERRY cautious about being excited here. My priors are like this: 1. Initial training of a neural network moves all weights around a large amount at first. 2. Later training of the network adjusts them a small amount. 3. An undertrained network will therefore look a lot like figuring out "positive, negative, or 0?" for each node during early training. If all these things are true, then 1. Early training of an…

Thank you. Your key point -- that so far all models with the proposed methods may have been only "grossly trained" -- is compelling. If I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-apples. That seems sensible to me, and makes replication easier, but I agree we need more to see extensive testing, after more extensi…

They also trained 3B with 2 trillion tokens.

> The number of training tokens is a crucial factor for LLMs. To test the scalability of BitNet b1.58 in terms of tokens, we trained a BitNet b1.58 model with 2T tokens following the data recipe of StableLM-3B [ TBMR], which is the state-of-the-art open-source 3B model.

> [..]

> Our findings shows that BitNet b1.58 achieves a superior performance on all end tasks, indicating that 1.58-bit LLMs also have strong generalization capabilities.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#234

Post by esafak »

Razengan wrote: post #206
gojomo wrote: post #177

That's not a 'bit' ("Binary digIT"). It's closer to a 'trit' ("TeRnary-digIT"). Specifically, ternary digits spanning {-1, 0, 1} (rather than the usual {0, 1, 2} in a base-3 numbering system) are 'balanced ternary'. A great intro to the theoretical reasons ternary might have some promise in computing is this 2001 article from 'American Scientist', "Third Base", which quotes Knuth calling balanced-ternary "perhaps the…

Why not a tit?

They renamed the biggest ML conference (NIPS) over the same joke, so don't count on it.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#235

Post by sigmoid10 »

jairuhme wrote: post #72
sigmoid10 wrote: post #50

Earlier quoted context omitted.

I'd be very surprised if Apple can put something on the level of GPT4 on a handheld. Remember, GPT4 is estimated to be around 1.7 trillion parameters. That's 3.4TB at 16 bit and it would still be ~340GB at 1.58bits. The best we can hope for is a low-ish level few billion parameter model. Which would still be cool on a phone, but as of today these models are nowhere near GPT4.

They won't have something at that size because as you pointed out, it is still huge. But depending on how they are used, smaller parameter models may be better for specific on-phone tasks that start to make the size of the model not a problem. GPT4 is so large because it is very general purpose with the goal seeming to be to answer anything. You could have a smaller model focused solely on Siri or something that woul…

The thing a about GPT4 that matters so much is not just raw knowledge retention, but complex, abstract reasoning and even knowing what it doesn't know. We haven't seen that yet in smaller models and it's unclear if it is even possible. The best we could hope for right now is a better natural language interface than Siri for calling OS functions.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#236

Post by mikewarot »

osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

>maybe NAND gates are not the ideal fundamental building block here?

It's my long held opinion that LUTs (Look Up Tables) are the basis of computation for the future. I've been pondering this for a long time since George Gilder told us that wasting transistors was the winning strategy. What could be more wasteful than just making a huge grid of LUTs that all interconnect, with NO routing hardware?

As time goes by, the idea seems to have more and more merit. Imagine a grid of 4x4 bit look up tables, each connected to its neighbors, and clocked in 2 phases, to prevent race conditions. You eliminate the high speed long lines across chips that cause so much grief (except the clock signals, and bits to load the tables, which don't happen often).

What you lose in performance (in terms of latency), you make up for with the homogenous architecture that is easy to think about, can route around bad cells, and be compiled to almost instantly, thanks to the lack of special cases. You also don't ever have to worry about latency, it's constant.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#237

Post by cs702 »

lr1970 wrote: post #163
cs702 wrote: post #150

Earlier quoted context omitted.

If I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-apples. That's sensible. It allows for easier replication of the results. Otherwise, I agree with you that more extensive testing, after more extensive pretraining, at larger model sizes, is still necessary.

towards the end of the paper they mentioned training on 2T tokens.

You're right. Thank you for pointing that out.
Post reply on HN