Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

321–330 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#321

Post by eru »

paul_mk1 wrote: post #281
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance…

You can probably apply the same techniques 'Deep neural networks are robust to weight binarization and other non-linear distortions' used to get to 0.68 bits / weight to get your ternary weights below one bit; so you can claim they are still one-bit networks.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#324

Post by esrauch »

Noe2097 wrote: post #271
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

There is another _shocking_ realization in this work: there are 11 types of people: those who know what binary means, those who don't, and those who say they do but actually don't. "The era of 1-bit LLMs" Representing { -1, 0, 1 } can't be done with 1-bit, I'm sorry -- and sad, please let's all get back to something vaguely sound and rigorous.

One trit but that's not a word anyone knows.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#325

Post by eru »

sandyarmstrong wrote: post #202
m3kw9 wrote: post #182

Earlier quoted context omitted.

How is this not lossy compression?

LLMs and vector embeddings are always lossy compression, yes?

Almost always. Though you can use them in a lossless compression system, too, with a few tricks.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#326

Post by eru »

jerf wrote: post #139
cs702 wrote: post #128

Earlier quoted context omitted.

> Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. I find it shocking that we don't even need lower floating-point precision. We don't need precision at all . We only need three symbols to represent every value. > I feel like this follows naturally from having only ternary values,…

An integer is just a concatenation of bits. Floating point appears more complicated but from an information theory perspective it is also just a concatenation of bits. If, for the sake of argument, one replaced a 64-bit int with 64 individual bits, that's really the same amount of information and a structure could hypothetically then either choose to recreate the original 64-bit int, or use the 64-bits more efficient…

> We humans have very stereotypical design methodologies and they have their advantages, but sometimes just letting algorithms rip can result in much better things than we could ever hope to design with the same resources.

Yes. Though here the interesting point is not so much that these structures exist, but that 'stupid' back-propagation is smart enough to find them.

You can't find busy beavers like that.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#327

Post by fabiospampinato »

w-m wrote: post #119

I was reading Exposing Floating Point today (as Airfoil is on the HN front page and I was perusing the archive of the author). It's a blog explaining the inner workings of floating point representations. About zero values it says [0]: > Yes, the floating point standard specifies both +0.0 and −0.0. This concept is actually useful because it tells us from which “direction” the 0 was approached as a result of storing v…

I would guess that having 2 zeros is not that useful for NNs, but in general with 2 bits we could encode 4 states, so are there 4 possible states that would be useful to encode? Sure, but would this be better than encoding 3 states? That's the entire question imo. I would guess that 3 states are probably better, because negative/neutral/positive seems the minimal signal that we need these weights to provide.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#328

Post by eru »

fzliu wrote: post #277
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

This will be big for FPGAs - adders are extremely cheap compared to multipliers and other DSP blocks.

Multipliers for eg 8 bit or 4 bit floating point values should also be pretty cheap? (I assume multipliers have a cost that grows quadratically with the number of bits?)

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#329

Post by eru »

verytrivial wrote: post #317
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

> If the proposed methods are implemented in hardware .. And the paper is _true_ of course, indeed, this sort of compounding quantum leap in efficiency due to representational change starts to get towards the Black Mirror / SciFi foundational mythology level of acceleration. Wild (if true!)

Slight tangent: in physics a quantum leap is the smallest possible change.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#330

Post by eru »

cooljoseph wrote: post #251
anon373839 wrote: post #137

Earlier quoted context omitted.

It’s a pity if realizing these gains absolutely requires full pre-training from scratch. I imagine more than a few people will at least try to find a way to repurpose the knowledge contained in existing models.

You can also have another model "mentor" a new model you are teaching to speed up training. You don't have to start from scratch with zero knowledge. This is done a lot in what are called distillations.

You can also re-use a lot of the infrastructure. Eg you can re-use your training data.
Post reply on HN