How does backprop work here? I can't imagine flipping bits of everything upstream of an error is effective.
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
81–90 of 462 posts
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#82Earlier quoted context omitted.
It's coming in October with the new Apple chip
I'd be very surprised if Apple can put something on the level of GPT4 on a handheld. Remember, GPT4 is estimated to be around 1.7 trillion parameters. That's 3.4TB at 16 bit and it would still be ~340GB at 1.58bits. The best we can hope for is a low-ish level few billion parameter model. Which would still be cool on a phone, but as of today these models are nowhere near GPT4.
You don't need "GPT4" though. Mixtral 8x7B is robust and can be run in 36 Gb, 24 Gb if you're willing to compromise. A 1.5 bit quantization should bring it down to 16. That's still a lot compared to the iPhone 15's 6, but it's close enough to imagine it happening soon. With some kind of streaming-from-flash architecture you might be in the realm already.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#83How does backprop work here? I can't imagine flipping bits of everything upstream of an error is effective.
(haven't read the paper). Maybe you can flip bits with a probability distribution that depends on the gradient?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#84After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I do…
After playing with OpenAI's GPT4 API, I'm quite convinced that LLMs would be in everything and everywhere today if inference cost is as low as loading a website and context size is 100x higher. In other words, only inference cost is holding it back from completely changing everything. So if we have a shortcut to getting something like GPT4 to run locally on a small device, watch out.
That is the plan. Even if these independent software improvements don't create 10x improvements NVDA and others are making huge improvements.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#85Also from Microsoft in 2021: Make Every feature Binary: A 135B parameter sparse neural network for massively improved search relevance [1]
[1] https://www.microsoft.com/en-us/research/blog/make-every-fea...
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#86Sooo, short Nvidia?
Hardly for this reason, but it does look suspiciously high doesn't it.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#87Any models published as well?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#88Interesting return to ternary. Effectively, each weight says only whether it's correlated (+1), uncorrelated (0), or anti-correlated (-1) with the input, and the structure of the network is the actual computation over that information.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#89How does backprop work here? I can't imagine flipping bits of everything upstream of an error is effective.
(haven't read the paper). Maybe you can flip bits with a probability distribution that depends on the gradient?
That's an interesting idea! Would love to try that on MNIST one day.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#90Powers of 3 don't pack well into binary memory... A 1 bit multiplier in silicon is a single logic gate, but a ternary decoder to decode a packed tri-state 'weight' is bigger. I therefore suspect that this method will be extended to make all weights simple 1 or 0 (ie. Binary). Perhaps that will be done by having half the weights have 1 or 0 values, while the other half are -1 or 0.
5 trits fit into 1 byte pretty well, since 3^5 = 243 is just under 2^8 = 256. That should be called an 8/5 = 1.6 bit model though, while the paper names it 1.58 bit, closer to log_2(3) ~ 1.5849625
Would be nice to have hardware instructions that work on 5 tris natively.