Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

341–350 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#341

Post by farhanhubble »

thesz wrote: post #283
mise_en_place wrote: post #191

Earlier quoted context omitted.

Intuitively I've always been a bit skeptical of quantization. Wouldn't there be a tiny loss in precision by doing this type of quantization? I could imagine the error function increasing by utilizing these types of techniques.

John Carmack pointed out (and I learned it here at HN) that what training really needs is the *sign" of each individual gradient parameter. I.e., you can quantize gradient to -1, 0 and 1 and still have neural network learn much of the dataset.

Wow! Is there a link to read up more on this?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#342

Post by eru »

nighthawk454 wrote: post #201
gojomo wrote: post #177

That's not a 'bit' ("Binary digIT"). It's closer to a 'trit' ("TeRnary-digIT"). Specifically, ternary digits spanning {-1, 0, 1} (rather than the usual {0, 1, 2} in a base-3 numbering system) are 'balanced ternary'. A great intro to the theoretical reasons ternary might have some promise in computing is this 2001 article from 'American Scientist', "Third Base", which quotes Knuth calling balanced-ternary "perhaps the…

Base e is the optimal base for number representation, so that’s probably why. Followed by base 3, then base 2. https://en.m.wikipedia.org/wiki/Radix_economy

Negative bases are probably even better, because you can represent negative numbers without worrying about extra sign handling.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#343

Post by ryeguy_24 »

How does gradient descent work with these discrete ternary parameters? If you compute the partial differential for a parameter, how do you determine what to nudge the parameter when updating on back propagation? Do you only update if the "nudging amount" meets a threshold?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#344

Post by Nevermark »

flockonus wrote: post #228
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Considering how much faster additions are processed, and how a particular silicon chip could be optimized for this very specific case; all parts added together perhaps could show >100x speed up vs current systems. I must concur, "wow".

For hardware, 2-argument ternary additions and multiplications should be very close in terms of the tiny circuit required for either.

If you are doing ternary calculations on 32/16-bit hardware, then the additions would be simpler.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#345

Post by Razengan »

esafak wrote: post #234
Razengan wrote: post #206

Earlier quoted context omitted.

Why not a tit?

They renamed the biggest ML conference (NIPS) over the same joke, so don't count on it.

Why? Just because it's spelled identical to a human body part?

This kind of shit is one of the most bizarre things about human society (or the prude cultures of it at least), to consider the most natural things so taboo and a "joke" to mention.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#346

Post by Animats »

Well, that's 2 bits, but still...

LLMs have gone from 32-bit floating point numbers down to 16 and 8 bit values. Now 2 bits. It's a hint as to how evolution did it. The basic component is simple and has very wide tolerances. There are just a lot of them. That's something biology can evolve.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#347

Post by theendisney »

jandrese wrote: post #154
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

It seems like the AI space is slowly coming back around to the old Thinking Machines CM-1 architecture. It's not too often in computing where you see ideas a full 40 years ahead of their time make it into production.

Memristors any moment now

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#348

Post by antimatter15 »

paul_mk1 wrote: post #281
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance…

They cite straight through estimators in the previous work with many of the same authors on (actual binary) BitNet

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#350

Post by fabmilo »

paul_mk1 wrote: post #281
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance…

They train using Straight Through Estimator but is cited in the previous BitNet paper. What happen to the TrueNorth Chip? I think investing in specialized hardware for AI is a good bet.
Post reply on HN