Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

401–410 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#401

Post by pclmulqdq »

edflsafoiewq wrote: post #397
bick_nyers wrote: post #392

Earlier quoted context omitted.

Extraordinary claims require extraordinary evidence. That's not to say that a 70B model is necessary, but surely something larger than 3B is doable, especially given that the results of the paper directly imply a significant reduction in memory requirements for training such a model.

> results of the paper directly imply a significant reduction in memory requirements for training such a model Isn't memory use in training higher, since they maintain high precision latent weights in addition to the binarized weights used in the forward pass?

Most research universities have the resources to train a ~10B parameter model, at least.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#402

Post by phkahler »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

>> we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value

Thinking out loud here. If you encode 64 weights in 2 64-bit words you can have the bits in one word indicating +1 if they're 1, and the bits in the other word indicating -1 if they are 1. You should be able to do the "products" with a few boolean operations on these 2 words to get a pair of 64 bit words for the result. Then summing becomes a matter of using a count-of-1's instruction on each word and subtracting the "negative" count from the positive. If AVX instructions can do this too, it seems like equivalent of 10-100 TOPS might be possible on a multi-core CPU.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#403

Post by nine_k »

mitthrowaway2 wrote: post #373
mjcohen wrote: post #360

Earlier quoted context omitted.

IIRC, Hamming's book "Digital Filters" (1989) has a section on FFTs with only the sign of the coefficient being used. It performed surprisingly well.

What is the sign of a complex number? Do you mean the phase?

AFAICT, both the real and imaginary components are from (-1, 0, +1) only. No single sign, but only 8 directions and the center.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#404

Post by TMWNN »

theendisney wrote: post #347
jandrese wrote: post #154

Earlier quoted context omitted.

It seems like the AI space is slowly coming back around to the old Thinking Machines CM-1 architecture. It's not too often in computing where you see ideas a full 40 years ahead of their time make it into production.

Memristors any moment now

I'm holding out for Josephson junctions

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#405

Post by imtringued »

jdthedisciple wrote: post #398

People have been doing this 6 years ago. https://github.com/yashkant/quantized-nets https://github.com/TropComplique/trained-ternary-quantization https://github.com/buaabai/Ternary-Weights-Network I too find it very interesting. But why this sudden, renewed fuzz?

Probably because despite the 1200 citations, they didn't have the ability to apply it to modern LLMs. Nobody cares about an image classifier using 50% less parameters since most of them were small enough to fit in memory anyway.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#406

Post by cs702 »

phkahler wrote: post #402
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

>> we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value Thinking out loud here. If you encode 64 weights in 2 64-bit words you can have the bits in one word indicating +1 if they're 1, and the bits in the other word indicating -1 if they are 1. You should be able to do the "products" with a few boolean operations…

Yes. More generally, this will enable implementation via crazy-cheap bit-wise ops in binary hardware, and possibly, maybe, via crazy-cheap trit-wise ops in ternary hardware that manipulates ternary digits, or trits. Note that any binary op over trits has only nine possible (trit, trit) input pairs and only three possible trit outputs. Maybe ternary hardware for AI will become a thing?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#407

Post by riskable »

smaddox wrote: post #355

Damn. Well, I guess I better hurry up and write and publish a paper on the Ternary Neural Network research that I've been doing (part-time) for the last several months, before it all gets scooped.

Modify your schedule, sure but do not rush it (just to beat the other folks). The first paper on any given topic may garner some 15 minutes of fame but the well-researched, boring paper is one oft-cited. Even if it isn't the first on its topic.

Be thorough and by golly, include some useful visuals! Even bad pictures and low-effort charts and graphs can vastly improve the grokability of a research paper.

Also, request assistance! Are you terrible at making charts and graphs? Ask someone to help you! For the low, low price of adding their name to the paper I'm 100% certain you can borrow an expert's time to add some dapper displays of useful information along with drastic wording and layout improvements.

The amount of papers in the wild that are just walls of jargon with completely useless, nearly-impossible-to-read charts and graphs is seemingly limitless.

Refreshing is the paper that a non-expert can read and understand! You don't have to ELI5 but well-written text and explanations are loved by all. The individual using it to gain actual knowledge will grok it from skimming and looking at the data anyway so you might as well take the time to explain some of the more complicated aspects like it's going to be read by a freshman STEM major (no need to go further back in education than that).

If you need help with grammar just paste a portion of your text into some LLM (even the small, locally-run models) and they usually do a pretty good job at finding and fixing such mistakes.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#408

Post by WiSaGaN »

paul_mk1 wrote: post #281
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance…

Could the reason that 3 states in this case be more efficient than 2 states be that 3 is closer to 2.718... (Euler's number) than 2 is?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#409

Post by cs702 »

cs702 wrote: post #128
creshal wrote: post #118

Earlier quoted context omitted.

> * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. Easier said than done, of course, and very impressive that they pulled it off. > In matr…

> Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. I find it shocking that we don't even need lower floating-point precision. We don't need precision at all . We only need three symbols to represent every value. > I feel like this follows naturally from having only ternary values,…

EDIT: Embarrassingly, on the last paragraph I got the number of possible input pairs wrong:

> only three possible input pairs, (a,b), (a,c), or (b,c) (within each pair, order doesn't matter)

The correct number, ignoring order, is six pairs, because we have to include (a,a), (b,b), and (c,c).

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#410

Post by intalentive »

StableAlkyne wrote: post #116
osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

It would be something of a full circle I feel went back to dedicated circuits for NNs - that's how they began life when Rosenblatt built his Perceptron. I remember reading a review on the history in grad school (can't remember the paper) where the author stated that one of the initial interests in NNs by the military was their distributed nature. Even back then, people realized you could remove a neuron or break a co…

The US military’s interest in network robustness led to the internet if I’m not mistaken.

Also preceding the perceptron was the McCulloch & Pitts neuron, which is basically a digital gate. NNs and computing indeed have a long history together.

Post reply on HN