Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

161–170 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#161

Post by cs702 »

jerf wrote: post #139
cs702 wrote: post #128

Earlier quoted context omitted.

> Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. I find it shocking that we don't even need lower floating-point precision. We don't need precision at all . We only need three symbols to represent every value. > I feel like this follows naturally from having only ternary values,…

An integer is just a concatenation of bits. Floating point appears more complicated but from an information theory perspective it is also just a concatenation of bits. If, for the sake of argument, one replaced a 64-bit int with 64 individual bits, that's really the same amount of information and a structure could hypothetically then either choose to recreate the original 64-bit int, or use the 64-bits more efficient…

> So from the perspective that it's all just bits in the end the only thing that is interesting is how useful it is to arrange those bits into trits for this particular algorithm, and that the algorithm seems to be able to use things more effectively that way than with raw bits.

Thank you. I find many other things interesting here, including the potential implications for hardware, but otherwise, yes, I agree with you, that is interesting.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#162

Post by nutanc »

cs702 wrote: post #158
nutanc wrote: post #122

Earlier quoted context omitted.

We have been experimenting with the paper( https://www.researchgate.net/publication/372834606_ON_NON-IT... ). There is a mathematical proof that binary representation is enough to capture the latent space. And in fact we don't even need to do "training" to get that representation. The practical application we tried out for this algorithm was to create an alternate space for mpnet embeddings of Wikipedia paragraphs. U…

You're talking about mapping floating-point vector representations, i.e., embeddings, computed by a pretrained LLM to binary vector representations, right? And you're talking about doing this by first having someone else 's pretrained LLM compute the embeddings, right? Sorry, but that seems only minimally, tangentially related to the topic of running LLMs in ternary space. I don't see how your comment is relevant to…

Yeah, sorry, needed a much bigger canvas than a comment to explain. Let me try again. The example I took was to show mapping from one space to another space and it may have just come across as not learning anything. Yes. You are right it was someone else's pretrained LLM. But this new space learnt the latent representations of the original embedding space. Now, instead of the original embedding space it could also have been some image representation or some audio representation. Even neural networks take input in X space and learn a representation in Y space. The paper shows that any layer of a neural network can in fact be replaced with a set of planes and we can represent a space using those planes and that those planes can be created in a non iterative way. Not sure if I am being clear, but have written a small blog post to show for MNIST how an NN creates the planes(https://gpt3experiments.substack.com/p/understanding-neural-...). Will write more on how once these planes are written, how we can use a bit representation instead of floating point values to get similar accuracy in prediction and next how we can draw those planes without the iterative training process.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#163

Post by lr1970 »

cs702 wrote: post #150
lr1970 wrote: post #148

Earlier quoted context omitted.

Authors reported perplexity only for small up to 3B weights models. On the other hand, they reported throughput for 70B model, but not its performance (perplexity, end-to-end tasks). Very unfortunate omission. Overall, the paper is rather poorly written.

If I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-apples. That's sensible. It allows for easier replication of the results. Otherwise, I agree with you that more extensive testing, after more extensive pretraining, at larger model sizes, is still necessary.

towards the end of the paper they mentioned training on 2T tokens.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#165

Post by rapatel0 »

The mathematics of the BNNs are sound. The shannon entropy of a word is really small (I vaguely remember ~2 bits). Also all neural networks are ridiculously over provisioned.

I worked on 7 years ago trying to efficiently binarize CNNs from existing models. It the difficult was getting training running without the losses going to high. I think that vision models will be much more difficult to binarize, but you might not need to with clip if the vision encoder stays in regular math {fp16,int8}

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#166

Post by [deleted] »

londons_explore wrote: post #47

Powers of 3 don't pack well into binary memory... A 1 bit multiplier in silicon is a single logic gate, but a ternary decoder to decode a packed tri-state 'weight' is bigger. I therefore suspect that this method will be extended to make all weights simple 1 or 0 (ie. Binary). Perhaps that will be done by having half the weights have 1 or 0 values, while the other half are -1 or 0.

[deleted]

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#167

Post by lovasoa »

rafaelero wrote: post #99
jart wrote: post #60

Earlier quoted context omitted.

LLMs will give normal people a firmer standing in technological society. That's a good thing. But will it change everything? Not a chance. Even if LLMs did change everything, that probably would not be a good thing. Dijkstra says Muslim algebra died when it returned to the rhetoric style, and the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters…

LLM's can do math as well.

- Hey ChatGTP ! What it 69*94 ?

- The result of 69*94 is 6466.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#169

Post by anon291 »

wongarsu wrote: post #36
lucubratory wrote: post #21

After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I do…

I wouldn't be surprised if this causes hardware startups to pop up that build accelerator cards tuned for this architecture. It seems stupidly simple to do inference in hardware, and with most of the training being quantized as well you might even be able to provide speedups (and energy savings) for training with reasonable investment and on cheaper processor nodes than what Nvidia is using. Sure, Nvidia might eat th…

There's like a million startups promising analog / bit-level computation, inference-only, cheap computation.

There's rain.ai, d-matrix, etc.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#170

Post by trebligdivad »

sigmoid10 wrote: post #56
osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

People are working on that [1]. In some sense, it's a step back to analog computing. Add/multiply is possible to do directly in memory with voltages, but it's less versatile (and stable) than digital computing. So you can't do all calculations in a neural network that way, meaning some digital components will always be necessary. But I'm pretty sure analog will make a comeback for AI chips sooner or later. [1] https:…

Trinary however is an interesting middle; people have built trinary hardware long ago; it feels like you could make natively trinary hardware for something like this; it might even be quite a win.
Post reply on HN