Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

221–230 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#221

Post by int_19h »

dns_snek wrote: post #120
vidarh wrote: post #117

Earlier quoted context omitted.

Last I checked (and confirmed by repeating it just now) GPT-4 did just fine at adding 2 numbers up, because it knows better now than to do that manually and will express it as Python. It does worse if you try to force it to do it step by step like a child and don't reinforce adherence to the rules every step, because just like humans it gets "sloppy" when you try to get it to repeat the same steps over and over. If y…

The claim was that "LLMs can do math". Below they linked a model from Google that might be capable of that, but as a general rule (and with OpenAI's models specifically) LLMs can't "do math" by any reasonable definition.

GPT-4 is a tiny subset of "LLMs".

If you train a model to do math (and optimize representation for that), it'll do math. GPT-4 just isn't, and, generally speaking, they aren't, because it's much more efficient to train them to "use a calculator". Same as with humans.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#222

Post by gliptic »

vessenes wrote: post #181
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

I'd be VERRY cautious about being excited here. My priors are like this: 1. Initial training of a neural network moves all weights around a large amount at first. 2. Later training of the network adjusts them a small amount. 3. An undertrained network will therefore look a lot like figuring out "positive, negative, or 0?" for each node during early training. If all these things are true, then 1. Early training of an…

> Also, a quick reminder that Perplexity 12 is really terrible.

The 3B model had a perplexity of 9.91, less than LLaMa 1 in fp16.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#223

Post by int_19h »

londons_explore wrote: post #52
sebzim4500 wrote: post #27

Earlier quoted context omitted.

These still run on GPUs

GPU's aren't yet awfully efficient at 1 bit math. I could imagine FPGA designs might be competitive. And dedicated ASIC's would almost certainly beat both by a decent margin.

I don't think it would be difficult to make them efficient.

The main reason why we run this stuff on GPUs is their memory bandwidth, anyway.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#224

Post by SkyBelow »

jerf wrote: post #139
cs702 wrote: post #128

Earlier quoted context omitted.

> Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. I find it shocking that we don't even need lower floating-point precision. We don't need precision at all . We only need three symbols to represent every value. > I feel like this follows naturally from having only ternary values,…

An integer is just a concatenation of bits. Floating point appears more complicated but from an information theory perspective it is also just a concatenation of bits. If, for the sake of argument, one replaced a 64-bit int with 64 individual bits, that's really the same amount of information and a structure could hypothetically then either choose to recreate the original 64-bit int, or use the 64-bits more efficient…

This sort of breakdown also reminds me of the explanation of why busy beavers grow faster than anything humans can ever define. Anything a human can define is a finite number of steps that can be represented by some turing machine of size M. A turning machine of size N > M can then use M as a subset of it, growing faster than than the turing machine of size M. Either it is the busy beaver for size N, or it grows slower than the busy beaver for size N. Either way, the busy beaver for size N grows faster than whatever the human defined that was captured by the turning machine of size M. This explanation was what helped me understand why busy beavers is faster growing than any operator that can be formally defined (obviously you can define an operator that references busy beaver itself, but busy beaver can be considered to not be formally defined, and thus any operator defined used it isn't formally defined either).

The bit about floating point numbers just being a collection of bits interpreted in a certain way helps make sense why a bigger model doesn't need floating points at all.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#225

Post by o11c »

nighthawk454 wrote: post #201
gojomo wrote: post #177

That's not a 'bit' ("Binary digIT"). It's closer to a 'trit' ("TeRnary-digIT"). Specifically, ternary digits spanning {-1, 0, 1} (rather than the usual {0, 1, 2} in a base-3 numbering system) are 'balanced ternary'. A great intro to the theoretical reasons ternary might have some promise in computing is this 2001 article from 'American Scientist', "Third Base", which quotes Knuth calling balanced-ternary "perhaps the…

Base e is the optimal base for number representation, so that’s probably why. Followed by base 3, then base 2. https://en.m.wikipedia.org/wiki/Radix_economy

FSVO "optimal". In practice, both physical reality and algorithm design strongly favors base 2.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#227

Post by nighthawk454 »

o11c wrote: post #225
nighthawk454 wrote: post #201

Earlier quoted context omitted.

Base e is the optimal base for number representation, so that’s probably why. Followed by base 3, then base 2. https://en.m.wikipedia.org/wiki/Radix_economy

FSVO "optimal". In practice, both physical reality and algorithm design strongly favors base 2.

Yeah, specifically, the definition of optimal provided - radix economy. There are plenty of other considerations one could make in other contexts. Practically, a transcendental base seems... rather impractical. And base 2 is not so much 'more optimal' than base 3 to warrant the electrical complexity probably, for example.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#228

Post by flockonus »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Considering how much faster additions are processed, and how a particular silicon chip could be optimized for this very specific case; all parts added together perhaps could show >100x speed up vs current systems.

I must concur, "wow".

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#229

Post by jonbaer »

yousif_123123 wrote: post #87

Any models published as well?

I really can't tell but it seems to be a continuation of this work if I read the To-Dos correctly, what do you think? Here it seems to be 1-bit on just the transformer, https://huggingface.co/shi3z/BitNetWikipedia110M

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#230

Post by cs702 »

vessenes wrote: post #181
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

I'd be VERRY cautious about being excited here. My priors are like this: 1. Initial training of a neural network moves all weights around a large amount at first. 2. Later training of the network adjusts them a small amount. 3. An undertrained network will therefore look a lot like figuring out "positive, negative, or 0?" for each node during early training. If all these things are true, then 1. Early training of an…

Thank you. Your key point -- that so far all models with the proposed methods may have been only "grossly trained" -- is compelling. If I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-apples. That seems sensible to me, and makes replication easier, but I agree we need more to see extensive testing, after more extensive pretraining, on models of larger sizes.
Post reply on HN