So we almost go back full circle to human (animal) brain binary spikes?
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
151–160 of 462 posts
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#152So for the uninitiated (me), does this mean the input is not a float (i.e. is quantized on input), such that all the math can be done with int operations?
This seems almost too good to be true.
Edit: Answering my own question, yes. The details are in the original bitnet paper: https://arxiv.org/abs/2310.11453
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#153There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…
I haven’t been keeping tabs, but this seems very much like RIP / Achilioptas version of the Johnson Lindenstrauss lemma.
Perhaps the rest of the JL lemma promise applies as well - compressing the number of parameters by a few orders of magnitude as well.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#154There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…
It seems like the AI space is slowly coming back around to the old Thinking Machines CM-1 architecture. It's not too often in computing where you see ideas a full 40 years ahead of their time make it into production.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#155I hope somebody gives this team access to the good data and a lot of crunch, I'd love to see what happens when you train the big fella.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#156I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?
People are working on that [1]. In some sense, it's a step back to analog computing. Add/multiply is possible to do directly in memory with voltages, but it's less versatile (and stable) than digital computing. So you can't do all calculations in a neural network that way, meaning some digital components will always be necessary. But I'm pretty sure analog will make a comeback for AI chips sooner or later. [1] https:…
Hadn't thought about it this way before, but given that LLMs are auto regressive (use their own data for next data), they're sensitive to error drift in ways that are rather similar to analog computers.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#157Earlier quoted context omitted.
After playing with OpenAI's GPT4 API, I'm quite convinced that LLMs would be in everything and everywhere today if inference cost is as low as loading a website and context size is 100x higher. In other words, only inference cost is holding it back from completely changing everything. So if we have a shortcut to getting something like GPT4 to run locally on a small device, watch out.
I'll agree with you, and add that inference speed is a big factor too. SDXL-ligtning/cascade can generate images in 200ms which is fast enough to fit in a web request, and paradoxically makes it even cheaper to generate. And using groq at 500 t/s is wild compared to any of the other platforms.
500 t/s is uncomfortably fast to me. Generating high quality answers at speeds faster than I can read is the point at which I feel like LLMs are magic.
I’m glad people are doing it though, and I’ll happily adapt to accessing inference at that speed.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#158There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…
We have been experimenting with the paper( https://www.researchgate.net/publication/372834606_ON_NON-IT... ). There is a mathematical proof that binary representation is enough to capture the latent space. And in fact we don't even need to do "training" to get that representation. The practical application we tried out for this algorithm was to create an alternate space for mpnet embeddings of Wikipedia paragraphs. U…
You're talking about mapping floating-point vector representations, i.e., embeddings, computed by a pretrained LLM to binary vector representations, right? And you're talking about doing this by first having someone else's pretrained LLM compute the embeddings, right? Sorry, but that seems only minimally, tangentially related to the topic of running LLMs in ternary space. I don't see how your comment is relevant to the discussion here.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#159There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…
> On existing hardware, the gains in compute and memory efficiency are significant, without performance degradation (as tested by the authors).
Did they actually show absence of performance degradation?
I think it's conspicuous that Table 1 and Table 2 in the paper, which show perplexity and accuracy results respectively, are only for small model sizes, whereas Figure 2, Figure 3 (latency, memory, energy consumption) and Table 3 (throughput) all show larger model sizes. So it seems like they had every opportunity to show the perplexity/accuracy comparisons at the larger model sizes, but did not include them.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#160There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…
> On existing hardware, the gains in compute and memory efficiency are significant, without performance degradation (as tested by the authors). Did they actually show absence of performance degradation? I think it's conspicuous that Table 1 and Table 2 in the paper, which show perplexity and accuracy results respectively, are only for small model sizes, whereas Figure 2, Figure 3 (latency, memory, energy consumption)…
Others have already made the same point in this thread. See my response here: https://news.ycombinator.com/item?id=39539508