Wondering if this might have any impact on the use of quantum computers in LLM training/distillation…
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
411–420 of 462 posts
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#412Earlier quoted context omitted.
>> we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value Thinking out loud here. If you encode 64 weights in 2 64-bit words you can have the bits in one word indicating +1 if they're 1, and the bits in the other word indicating -1 if they are 1. You should be able to do the "products" with a few boolean operations…
Yes. More generally, this will enable implementation via crazy-cheap bit-wise ops in binary hardware, and possibly, maybe, via crazy-cheap trit-wise ops in ternary hardware that manipulates ternary digits, or trits. Note that any binary op over trits has only nine possible (trit, trit) input pairs and only three possible trit outputs. Maybe ternary hardware for AI will become a thing?
Fleshing out my thought above. If we want to multiply A*B = C and all operands are stored in 2 separate bits Ap and An (Ap = 1 if A = +1 while An = 1 if A = -1). We can do a product with:
Cp = (Ap & Bp) | (An & Bn)
Cn = (An & Bp) | (Ap & Bn)
So 64 products in 6 instructions, or 256 in 6 instructions with AVX2, or 512 in six instructions using AVX512. If you can execute 2 instructions at a time on different words, this becomes 1024 "products" in 6 cycles or between 0.5 and 1 TOP per core.
The summing still involves using popcount on the positive and negative bits - I doubt AVX supports that but its still a fast way to "sum" individual bits. I don't see custom hardware for this as a short term thing - they need to prove out the quantization concept more first.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#413Earlier quoted context omitted.
Yes. More generally, this will enable implementation via crazy-cheap bit-wise ops in binary hardware, and possibly, maybe, via crazy-cheap trit-wise ops in ternary hardware that manipulates ternary digits, or trits. Note that any binary op over trits has only nine possible (trit, trit) input pairs and only three possible trit outputs. Maybe ternary hardware for AI will become a thing?
Fleshing out my thought above. If we want to multiply A*B = C and all operands are stored in 2 separate bits Ap and An (Ap = 1 if A = +1 while An = 1 if A = -1). We can do a product with: Cp = (Ap & Bp) | (An & Bn) Cn = (An & Bp) | (Ap & Bn) So 64 products in 6 instructions, or 256 in 6 instructions with AVX2, or 512 in six instructions using AVX512. If you can execute 2 instructions at a time on different words, thi…
> I don't see custom hardware for this as a short term thing - they need to prove out the quantization concept more first.
Yes, I agree. This still needs to be more extensively tested.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#414Is there any rigorous way to answer the question of how much information (be it entropy or some other measurement) is contained in a model's weights?
Yes, actually: That's the entire point of the paper! The concept is that the amount of information contained in a weight like 0.00006103515625 is equivalent to 0. -0.99951172 is equivalent to -1, 1.26406236 equivalent to 1, etc. That there's no practical difference when actually utilizing the model (if trained in ternary from the start).
The paper posits (and provides evidence) that if you train a model using ternary values instead of floating point values you get equivalent (useful/practical) information. You can't take an existing model and round all the values down to `{-1,0,+1}` values but you can (re)train a model using ternary values to get the same end result (equivalent information/output).
Technically a model trained using FP16 values contains vastly more information than a model trained using ternary values. Practically though it seems to make no difference.
My prediction: Floating point models will still be used extensively by scientists and academics in their AI research but nearly all real-world, publicly-distributed AI models will be ternary. It's just too practical and enticing! Even if the ternary representation of a model is only 90% effective it's going to be so much faster and cheaper to use it in reality. We're talking about the difference between requiring a $500 GPU or a $5 microcontroller.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#415These models will are compatible with llama.cpp out of the box, we (GigaML - https://gigaml.com ) are planning to train a small model (3-4B, 1-bit, opensource) with the latest stack-v2 dataset released today. Let me know if anyone is interested in collaborating with us.
[deleted]
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#416Earlier quoted context omitted.
You do realize that arithmetic is a very simple symbolic manipulation task? All you have to do is keep track of the carry. I haven't seen an LLM that couldn't get digit by digit addition done, but they always mess up the carry.
Just like humans. Try to get regular people do e.g. add 15-16 digit numbers (where is typically where I'd see GPT4 start to get "sloppy" unless you prompt it the way you would a child who's learning and is still prone to get annoyed and wonder why the hell you make them to it manually), and see how many start making mistakes. I find it really comical that this is what people complain about GPT over - there's zero ben…
this goes into the heart of what it means to "know".
All human knowledge is "symbolic". that is, knowledge is a set of abstractions (concepts) along with relations between concepts. As an example, by "knowing" addition is to understand the "algorithm" or operations involved in adding two numbers. reasoning is the act of traversing concept chains.
LLMs dont yet operate at the symbolic level, and hence, it could be argued that they dont know anything. LLM is a modern sophist excelling at language but not at reasoning.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#417Earlier quoted context omitted.
> the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters of medieval scholasticism I can propose an alternate view of things. Not that I'm going to argue that it is the only true statement in the world, but I think it is necessary for a thought to progress to have an alternative hypothesis. So the proposition is: formal symbolisms can deal only wi…
> So the proposition is: formal symbolisms can deal only with those problems that where already solved in imprecise human's languages. > To invent calculus and orbital mechanics you need first to talk for a several centuries (or thousands of years?) about what is position and velocity, you need to talk your way upto acceleration, and then you need to find a way to measure them and to define in a strict geometric term…
> all kinds of problems that you would never in a thousand years be able to handle with mere talk
Not "all kinds of problems" but very specific kinds of problems which is possible to formalize into a math language. How would you go about inventing thermodynamics if you didn't know words "temperature" and "pressure"? You'd need to start for your senses that can tell you "this is a hot surface", or "this is a cold one", or "this one is colder than that", you need to decide that "coldness" is a "negative heat" (it is not the most obvious idea for an animal, because animals have as receptors for a cold, so receptors for a heat, you could feel hot and cold at the same time, if you managed to stimulate both kinds of receptors at the same time). Then you need to notice that some materials change volume when heated, then you need to come up with an idea to use measurements of a volume to measure a temperature, and only then you can try to invent pV=nRT, which becomes almost tautological at that point, because your operational definition of a temperature makes it equivalent to a volume.
After that you really can use calculus and make all sorts of quantitative statements about thermodynamic systems. But before all that "mere talk" was finished thermodynamics was not a kind of a problem calculus can deal with.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#418Is there anything about this specific to LLMs, or could you use it for any transformer based model? It seems like they made a modified transformer.
It seems like it could be any transformer, which is exciting now that even in imaging gradient transformers are all the rage. But ideally we'd need to see this result in other transformers (but I have a hard time seeing why it wouldn't be the case).
At the very least it could be used to reduce the requirements and speed up the prompt recognition step(s) of image-based generative AI.
"Stable Diffusion 3 XS" will use ternary? Here's to hoping :)
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#419Earlier quoted context omitted.
Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance…
Could the reason that 3 states in this case be more efficient than 2 states be that 3 is closer to 2.718... (Euler's number) than 2 is?
Why not have some layers/nodes/systems be 2 states and have others be 3... couldn't you get arbitrarily close to Euler's number that way?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#420There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…
It almost seems too good to be true