Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

301–310 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#301

Post by schiffern »

dekhn wrote: post #205
gojomo wrote: post #177

That's not a 'bit' ("Binary digIT"). It's closer to a 'trit' ("TeRnary-digIT"). Specifically, ternary digits spanning {-1, 0, 1} (rather than the usual {0, 1, 2} in a base-3 numbering system) are 'balanced ternary'. A great intro to the theoretical reasons ternary might have some promise in computing is this 2001 article from 'American Scientist', "Third Base", which quotes Knuth calling balanced-ternary "perhaps the…

How useful are -0 and 0? You could splurge on two bits per value which gives you { -1, -0, 0, 1 }

Rather than (indistinguishable?) 0 and -0, why not add back some magnitude in the positive direction?

  { -1, 0, 1, 2 }
is most obvious, but it's not clear whether it's better or worse than

  { -1, 0, 1/2, 1 }
Maybe theoretically (if not architecturally) it would best to "split the difference" between the two and choose

  { -1, 0, 1/phi, phi }
or perhaps the more implementable

  { -1, 0, 1, 3 }
EDIT: Of course you can also go the other way, with

  { -1, 1 }

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#302

Post by nxobject »

paul_mk1 wrote: post #281
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance…

As aside, I'm curious: what was it like to work at IBM research, especially as a legacy industrial research org?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#303

Post by dekhn »

schiffern wrote: post #301
dekhn wrote: post #205

Earlier quoted context omitted.

How useful are -0 and 0? You could splurge on two bits per value which gives you { -1, -0, 0, 1 }

Rather than (indistinguishable?) 0 and -0, why not add back some magnitude in the positive direction? { -1, 0, 1, 2 } is most obvious, but it's not clear whether it's better or worse than { -1, 0, 1/2, 1 } Maybe theoretically (if not architecturally) it would best to "split the difference" between the two and choose { -1, 0, 1/phi, phi } or perhaps the more implementable { -1, 0, 1, 3 } EDIT: Of course you can also g…

-0 is not indistinguishable from 0 in floating point math. Most ops return +0 and -0 can behave differently. I don't know of any examples where -0 is important for machine learning, though.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#304

Post by llm_trw »

arunk47 wrote: post #258
llm_trw wrote: post #39

Earlier quoted context omitted.

Depends. The only paper they cite for training: https://arxiv.org/pdf/2310.11453.pdf doesn't improve training costs much and most models are already training constrained. Not everyone has $200m to throw at training another model from scratch.

Is there any scope for indie builders?

Not really. These are slightly better for memory during pre-training and fine turning but not enough to make a 4090 usable even for a 7b model.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#305

Post by Solvency »

thesz wrote: post #283
mise_en_place wrote: post #191

Earlier quoted context omitted.

Intuitively I've always been a bit skeptical of quantization. Wouldn't there be a tiny loss in precision by doing this type of quantization? I could imagine the error function increasing by utilizing these types of techniques.

John Carmack pointed out (and I learned it here at HN) that what training really needs is the *sign" of each individual gradient parameter. I.e., you can quantize gradient to -1, 0 and 1 and still have neural network learn much of the dataset.

Why isn't John Carmack working for OpenAI? Hell, why did he waste years at Meta to work on a VR headset and NOT AI? He even announced he wants to focus on AGI but he missed out on literally all the action.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#307

Post by Solvency »

bilsbie wrote: post #238

This really just sounds absurd. How can ternary possibly encode enough information? Anyone willing to explain it like I’m a Django developer who watched half a karpathy video?

Because by making the model larger you don't need 64bit precision floats you only need 64 discrete bits.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#308

Post by WhitneyLand »

paul_mk1 wrote: post #281
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance…

That’s really interesting to see the breadcrumb trail goes back that far.

So what are the most important insights in this paper compared to what was previously done?

I assume there’s more context to the story and it’s not just that no one thought to apply the concepts to LLM’s until now?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#309

Post by refulgentis »

ignoramous wrote: post #65
anon373839 wrote: post #5

> BitNet b1.58 can match the performance of the full precision baseline starting from a 3B size. ... This demonstrates that BitNet b1.58 is a Pareto improvement over the state-of-the-art LLM models. > BitNet b1.58 is enabling a new scaling law with respect to model performance and inference cost. As a reference, we can have the following equivalence between different model sizes in 1.58-bit and 16-bit based on the re…

I wonder if 1bit quantization is the main reason why pplx.ai is faster than any other RAG or chatbot. For instance, Gemini in comparison is a turtle, though it is better at explanations, while pplx is concise.

Absolutely not, 1 bit isn't even real yet. perplexity does a ton of precaching, TL;Dr every novel query is an opportunity to cache: each web page response, the response turned into embeddings, and the LLM response. That's also why I hate it, it's just a rushed version of RAG with roughly the same privacy guarantees any incumbent would have given you in last 15 years (read: none, and gleefully will exploit yours while saying "whoops!")

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#310

Post by fnordpiglet »

cooljoseph wrote: post #251
anon373839 wrote: post #137

Earlier quoted context omitted.

It’s a pity if realizing these gains absolutely requires full pre-training from scratch. I imagine more than a few people will at least try to find a way to repurpose the knowledge contained in existing models.

You can also have another model "mentor" a new model you are teaching to speed up training. You don't have to start from scratch with zero knowledge. This is done a lot in what are called distillations.

This came out a little bit ago, my open question is if this approach can be used to port weights between architectures like this.

https://arxiv.org/abs/2402.13144

Post reply on HN