If this paper (especially the results on Table 4) is true, then this is a game changer!
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
41–50 of 462 posts
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#42So are there any details on the algorithms they used for backprop? I'm not seeing any in the paper other than "we used a lot of tokens".
Does this help? https://arxiv.org/abs/2310.11453
It seems to have more details (it's the paper before the linked one) about the actual training, but I'm scanning it and this isn't my field so maybe it's too light also.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#43So are there any details on the algorithms they used for backprop? I'm not seeing any in the paper other than "we used a lot of tokens".
It's a fairly straightforward modification of BitNet, so I assume this quote from the BitNet paper applies:
To train our 1-bit model, we employ the straight-through estimator (STE)[BLC13 ] to approximate the gradient during backpropagation. This method bypasses the non-differentiable functions, such as the Sign (Eq. 2) and Clip (Eq. 5) functions, during the backward pass. STE allows gradients to flow through the network without being affected by these non-differentiable functions, making it possible to train our quantized model
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#44I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#45Can someone versed in the ways of math explain how this is different from previous quantization methods? And specifically, seeing how going from 16fp to 8bit mostly gives same perplexity while anything further seems to lose quality / dumb down the model, how is this even less precise method is able to achieve this?
If I understand it correctly, this seems to be more than just quantizing, the models are apparently trained in this format as well. So it's possible that the many layers adjust themselves in a way that "cancels out" the inaccuracies of the lower bit count
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#46Shouldn’t that be “1-trit”?
Read the pdf https://arxiv.org/pdf/2402.17764.pdf they call it 1-bit everywhere. I don't know why do they do this, 1-bit seems to be a very wrong name for {-1, 0, 1}.
I think 0 "doesn't count", since you don't have to add or subtract anything for it, just mask it out.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#47Powers of 3 don't pack well into binary memory...
A 1 bit multiplier in silicon is a single logic gate, but a ternary decoder to decode a packed tri-state 'weight' is bigger.
I therefore suspect that this method will be extended to make all weights simple 1 or 0 (ie. Binary). Perhaps that will be done by having half the weights have 1 or 0 values, while the other half are -1 or 0.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#48This is great, my employer just gave me a M1 laptop with only 16gb ram and I had to downgrade my 7B parameter local LLM’s to 3 bit quantizing, they’ve been surprisingly okay! In my personal machine at 64gb ram, I usually use 8x7B at Q5 or 70B at Q4 Its Mistral all the way down! Imagining Q1.58 that’s doing well makes me happy
Quantized 7B LLMs should work fine on your machine, though maybe you’re talking about speed?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#49when can we expect the first ~100+ million parameter models to run on raspberry pi Pico?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#50Earlier quoted context omitted.
After playing with OpenAI's GPT4 API, I'm quite convinced that LLMs would be in everything and everywhere today if inference cost is as low as loading a website and context size is 100x higher. In other words, only inference cost is holding it back from completely changing everything. So if we have a shortcut to getting something like GPT4 to run locally on a small device, watch out.
It's coming in October with the new Apple chip
I'd be very surprised if Apple can put something on the level of GPT4 on a handheld. Remember, GPT4 is estimated to be around 1.7 trillion parameters. That's 3.4TB at 16 bit and it would still be ~340GB at 1.58bits. The best we can hope for is a low-ish level few billion parameter model. Which would still be cool on a phone, but as of today these models are nowhere near GPT4.