Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

371–380 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#371

Post by thesz »

farhanhubble wrote: post #341
thesz wrote: post #283

Earlier quoted context omitted.

John Carmack pointed out (and I learned it here at HN) that what training really needs is the *sign" of each individual gradient parameter. I.e., you can quantize gradient to -1, 0 and 1 and still have neural network learn much of the dataset.

Wow! Is there a link to read up more on this?

  > It is interesting that things still train even when various parts are pretty wrong — as long as the sign is right most of the time, progress is often made.
https://forums.fast.ai/t/how-to-do-reproducible-models-and-u...

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#372

Post by bjornsing »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

> * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the paper for exact details.

Aren’t you over complicating it a bit here? A dot product between a vector of activations (a₁, a₂, …) and a vector of ternary weights (b₁, b₂, …) can of course be computed as the sum of all activations for which the weight is 1, minus the sum of all activations for which the weight is -1.

It can’t however be computed as (a₁+b₁ + a₂+b₂ ...). You must have gotten that wrong.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#373

Post by mitthrowaway2 »

mjcohen wrote: post #360
paul_mk1 wrote: post #281

Earlier quoted context omitted.

Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance…

IIRC, Hamming's book "Digital Filters" (1989) has a section on FFTs with only the sign of the coefficient being used. It performed surprisingly well.

What is the sign of a complex number? Do you mean the phase?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#374

Post by gemeral »

Solvency wrote: post #307
bilsbie wrote: post #238

This really just sounds absurd. How can ternary possibly encode enough information? Anyone willing to explain it like I’m a Django developer who watched half a karpathy video?

Because by making the model larger you don't need 64bit precision floats you only need 64 discrete bits.

Do you mind pointing out where they make the model larger? The paper seems to suggest they are maintaining the same model sizes.

> Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#375

Post by kaelwd »

Razengan wrote: post #345
esafak wrote: post #234

Earlier quoted context omitted.

They renamed the biggest ML conference (NIPS) over the same joke, so don't count on it.

Why? Just because it's spelled identical to a human body part? This kind of shit is one of the most bizarre things about human society (or the prude cultures of it at least), to consider the most natural things so taboo and a "joke" to mention.

https://youtu.be/VS11bRa8NnQ?t=122

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#376

Post by baq »

eru wrote: post #332
osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

Analog computing for neural networks is always very tempting. > We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Transistors that are fully closed or fully open use basically no energy: they either have approximately zero current or approximately zero resistance. Transistors that are partially open dissipate a lot of energ…

As someone who knows almost nothing about electronics I assume you’d want a transistor which can open in two ways: with positive and negative voltage. I’ve seen TNAND built out of normal transistors, not sure if such exotic ones would help even if they were physically possible.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#377

Post by rfoo »

cs702 wrote: post #143
jxy wrote: post #141

Earlier quoted context omitted.

The matrices (weights) are ternary. The vectors are not.

The activations are in (-1, 1), so they're also representable by (-1, 0, 1).

This is wrong. The paper described that their activation is in int8 during inference.

That being said, before-LLM-era deep learning already had low bit quantization down to 1w2f [0] working back in 2016 [1]. So it's certainly possible it would work for LLM too.

[0] 1-bit weights, 2-bit activations; though practically people deployed 2w4f instead. [1] https://arxiv.org/abs/1606.06160

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#378

Post by tarruda »

arunk47 wrote: post #269
Klipper3 wrote: post #51

The theoretical capacity of a binary network is 69% of the capacity of a full-weight network, so it makes sense that LLM would converge to 1-bit networks in the long term. It's nice to finally see practical networks reach the theoretical limits found in the statistical mechanics of Ising models. A good pointer to efficient 1-bit training, from the statistical mechanics point of view, is here: https://www.pnas.org/doi…

What is stopping us right now from doing this one bit networks ?

I think no code was released yet

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#379

Post by igleria »

Solvency wrote: post #305
thesz wrote: post #283

Earlier quoted context omitted.

John Carmack pointed out (and I learned it here at HN) that what training really needs is the *sign" of each individual gradient parameter. I.e., you can quantize gradient to -1, 0 and 1 and still have neural network learn much of the dataset.

Why isn't John Carmack working for OpenAI? Hell, why did he waste years at Meta to work on a VR headset and NOT AI? He even announced he wants to focus on AGI but he missed out on literally all the action.

he has his own AGI startup now https://dallasinnovates.com/john-carmacks-keen-technologies-...

TBH I think they won't get anywhere. Doing good game engine work... why that would translate to AGI?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#380

Post by creshal »

threatripper wrote: post #284
pennomi wrote: post #270

Earlier quoted context omitted.

You can pack two trits into three bits, however. So one byte could hold 5 values instead of 4.

How exactly would you do that? 3 states need 1.58 bits which is a tad more than 1.5. Two 3-states have 3²=9 states while three bits only give you 2³=8 states.

I wonder if there's some encoding tricks you can use to reduce it to 8 (or less?) effective states, given that you're only using them with a reduced set of mathematical operations. E.g., can you automatically convert all (-1, 1) to (1, -1) and save one encoded state, since they add up to the same result anyway?
Post reply on HN