Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

281–290 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#281

Post by paul_mk1 »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there).

Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work).

Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance this will work for LLMs too.

https://arxiv.org/abs/1606.01981

This was a passion project of mine in my last few months at IBM research :).

I am convinced there is a deep connection to understanding why backprop is unreasonably effective, and the result that you can train low precision DNNs; for those note familiar, the technique is to compute the loss wrt to the low precision parameters (eg project to ternary) but apply the gradient to high precision copy of parameters (known as the straight through estimator). This is a biased estimator and there is no theoretical underpinning for why this should work, but in practice it works well.

My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#282

Post by m00x »

eigenvalue wrote: post #279

Is it really so surprising that something like this works given how human brain neurons work? My admittedly basic understanding is that these operate through an all-or-nothing principle for their action potentials (firing): they either fire or they don't, based on whether the input signals reach a certain threshold. So the output is already sort of binary in biological neurons. The inputs are more like continuous val…

This isn't really how neurons work.

First of all, they operate independent of a synchronized clock, and they can also accumulate signals instead of executing on a input. Neuromorphic chips are closer to how the brain works, but they're still super early. I believe Intel has the best one with the Loihi 2.

(Not a neuroscientist but my wife is and that's what I understand from our chats)

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#283

Post by thesz »

mise_en_place wrote: post #191
vessenes wrote: post #181

Earlier quoted context omitted.

I'd be VERRY cautious about being excited here. My priors are like this: 1. Initial training of a neural network moves all weights around a large amount at first. 2. Later training of the network adjusts them a small amount. 3. An undertrained network will therefore look a lot like figuring out "positive, negative, or 0?" for each node during early training. If all these things are true, then 1. Early training of an…

Intuitively I've always been a bit skeptical of quantization. Wouldn't there be a tiny loss in precision by doing this type of quantization? I could imagine the error function increasing by utilizing these types of techniques.

John Carmack pointed out (and I learned it here at HN) that what training really needs is the *sign" of each individual gradient parameter. I.e., you can quantize gradient to -1, 0 and 1 and still have neural network learn much of the dataset.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#284

Post by threatripper »

pennomi wrote: post #270
SushiHippie wrote: post #189

Earlier quoted context omitted.

But if I understand it correctly, they already need to use 2 bits, one for the sign and another one for the value, so there is already one wasted state, which could be used for -0.

You can pack two trits into three bits, however. So one byte could hold 5 values instead of 4.

How exactly would you do that? 3 states need 1.58 bits which is a tad more than 1.5. Two 3-states have 3²=9 states while three bits only give you 2³=8 states.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#285

Post by giantrobot »

jandrese wrote: post #154
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

It seems like the AI space is slowly coming back around to the old Thinking Machines CM-1 architecture. It's not too often in computing where you see ideas a full 40 years ahead of their time make it into production.

IIUC the main issue with the CM-1 architecture was feeding the processor cluster with data. That required a heftier front end system than was practical/affordable at the time. With modern CPUs and memory subsystems the GPUs can be saturated pretty easily. So going back to huge clusters of super narrow cores won't starve them for work.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#287

Post by npunt »

w-m wrote: post #133
rfoo wrote: post #123

Earlier quoted context omitted.

Interesting, how do you use -0 in the add, then? Is -0+1-1 a 0 or a -0? > Could the additional -0 carry some pseudo-gradient information It looks like training was done on fp32 or bf16. Low-bit quantization is approximated with STE during training. I'd expect training itself cause each point to "polarize" towards 1 or -1. > 2-bit quantizations being proposed Symmetric (i.e. without 0) exponential values were pretty p…

> how do you use -0 in the add In my mind the two zero values would represent a tiny epsilon around 0, let's say -0.01 and +0.01. Looking at them like this, it would mean +0 +0 -0 = +0 +0 -0 -0 = -0 +1 * +0 = +0 -1 * +0 = -0 Performing addition with the same sign count in each group would be problematic. How to decide on the sign of +0-0 or +1-1, other than flipping a coin?

maybe they could be stored together in two words until they're operated on and lose their pairing?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#289

Post by gradascent »

vessenes wrote: post #181
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

I'd be VERRY cautious about being excited here. My priors are like this: 1. Initial training of a neural network moves all weights around a large amount at first. 2. Later training of the network adjusts them a small amount. 3. An undertrained network will therefore look a lot like figuring out "positive, negative, or 0?" for each node during early training. If all these things are true, then 1. Early training of an…

Then perhaps a method emerges out of this to make training faster (but not inference) - do early training on highly quantized (even ternary) weights, and then swap out the weights for fp16 or something and fine-tune? Might save $$$ in training large models.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#290

Post by pests »

nutanc wrote: post #162
cs702 wrote: post #158

Earlier quoted context omitted.

You're talking about mapping floating-point vector representations, i.e., embeddings, computed by a pretrained LLM to binary vector representations, right? And you're talking about doing this by first having someone else 's pretrained LLM compute the embeddings, right? Sorry, but that seems only minimally, tangentially related to the topic of running LLMs in ternary space. I don't see how your comment is relevant to…

Yeah, sorry, needed a much bigger canvas than a comment to explain. Let me try again. The example I took was to show mapping from one space to another space and it may have just come across as not learning anything. Yes. You are right it was someone else's pretrained LLM. But this new space learnt the latent representations of the original embedding space. Now, instead of the original embedding space it could also ha…

> how we can draw those planes without the iterative training process.

Sounds interesting, but this is the part I would need more explanation on.

Just started reading your linked blog, I see it goes into some details there.

Post reply on HN