241–250 of 462 posts
Could there be some value in recognizing areas where the model needs finer grained weights and somehow using a different data type just in certain areas?
Earlier quoted context omitted.
You don't need "GPT4" though. Mixtral 8x7B is robust and can be run in 36 Gb, 24 Gb if you're willing to compromise. A 1.5 bit quantization should bring it down to 16. That's still a lot compared to the iPhone 15's 6, but it's close enough to imagine it happening soon. With some kind of streaming-from-flash architecture you might be in the realm already.
> With some kind of streaming-from-flash architecture you might be in the realm already. I thought mmap'ing models to only keep the currently needed pieces in RAM was something that was figured out ~6 months ago? Performance wasn't terribly great iirc, but with how much faster 1.58B is, it should still be okay-ish.
How is it a 1 bit LLM if 2 bits are required for each weight (and one of the 4 possible states is wasted to be able to represent 0)
This really just sounds absurd. How can ternary possibly encode enough information? Anyone willing to explain it like I’m a Django developer who watched half a karpathy video?
This is exciting news, if the 8B numbers are true, we can already use model like Mixtral 8x7, even with a single GPU? But further into the development, we need comparison to large model sizes. 70B might be too much to ask, but 13B should be there at least.
Earlier quoted context omitted.
Because it's no longer a linear optimization or curve fitting problem. It becomes a voting or combinatorial problem. Which at least in my mind are two completely different areas of research.
With enough parameters, it probably starts looking continuous again. Like how in physics everything is quantised at the smallest scale but if you put enough atoms together it all smooths out and behaves "classically".
Prior art: Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1 https://arxiv.org/abs/1602.02830 Ternary Neural Networks for Resource-Efficient AI Applications https://arxiv.org/abs/1609.00222
There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…
We have been experimenting with the paper( https://www.researchgate.net/publication/372834606_ON_NON-IT... ). There is a mathematical proof that binary representation is enough to capture the latent space. And in fact we don't even need to do "training" to get that representation. The practical application we tried out for this algorithm was to create an alternate space for mpnet embeddings of Wikipedia paragraphs. U…
I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?
>maybe NAND gates are not the ideal fundamental building block here? It's my long held opinion that LUTs (Look Up Tables) are the basis of computation for the future. I've been pondering this for a long time since George Gilder told us that wasting transistors was the winning strategy. What could be more wasteful than just making a huge grid of LUTs that all interconnect, with NO routing hardware? As time goes by, th…