Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

361–370 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#361

Post by p1esk »

int_19h wrote: post #352
p1esk wrote: post #318

Earlier quoted context omitted.

Are you sure? I was under impression that 3b quantization still results in a significant degradation. Which quantization method are you talking about?

It does result in a significant degradation relative to unquantized model of the same size , but even with simple llama.cpp K-quantization, it's still worth it all the way down to 2-bit. The chart in this llama.cpp PR speaks for itself: https://github.com/ggerganov/llama.cpp/pull/1684#issue-17396...

Oh wow, you’re right. Though it seems that they are using very small weight group sizes: either 16 or 32 (fp16 scaling factor per group). In this paper it seems there’s no weights grouping, so it’s a bit apples to oranges.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#362

Post by lucubratory »

joelthelion wrote: post #78

Assuming this is confirmed, what's the impact on training? Inference is definitely an issue for LLMs right now. But if training were suddenly possible for lone hackers (or maybe smaller companies), it would open up a lot of new possibilities as well.

In theory it should make training a lot easier too, particularly on CPUs. But I think you'll still need reasonably expensive compute to get a model something close to the current big models, and you really can't ignore data. Data quality and quantity are both huge ingredients in model quality, at least as big as architecture. It's still non-trivial to get a good quality, large dataset, certainly out of the reach of lone hackers and most small companies.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#363

Post by vitorgrs »

ignoramous wrote: post #65
anon373839 wrote: post #5

> BitNet b1.58 can match the performance of the full precision baseline starting from a 3B size. ... This demonstrates that BitNet b1.58 is a Pareto improvement over the state-of-the-art LLM models. > BitNet b1.58 is enabling a new scaling law with respect to model performance and inference cost. As a reference, we can have the following equivalence between different model sizes in 1.58-bit and 16-bit based on the re…

I wonder if 1bit quantization is the main reason why pplx.ai is faster than any other RAG or chatbot. For instance, Gemini in comparison is a turtle, though it is better at explanations, while pplx is concise.

Nop. The model on Perplexity is a finetuned GPT 3.5 (the free one). And the paid versons, well, you can choose between GPT4 (not turbo), Gemini pro, Claude, etc.

You can choose their model ("Experimental"), but is not faster than the other models.

All of these, proprietary models are fast on Perplexity. I do guess they are using some insane cache system, better API infrastructure...

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#364

Post by singularity2001 »

concrete_head wrote: post #197
singularity2001 wrote: post #151

So we almost go back full circle to human (animal) brain binary spikes?

It's not quiet spikes but getting closer to the idea. I'm amazed it has taken this long for this type of thing to reach HN which gives next to no attention to spiking neural networks. Simon Thorpe, a CNRS researcher has got some fascinating papers and lectures on YouTube on using binary weights on neuromorphic hardware which has had practical applications for over 20 years already. I made an account just to drop his…

why is his name so dangerous you can't drop it on your main account lel?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#365

Post by edflsafoiewq »

ryeguy_24 wrote: post #343

How does gradient descent work with these discrete ternary parameters? If you compute the partial differential for a parameter, how do you determine what to nudge the parameter when updating on back propagation? Do you only update if the "nudging amount" meets a threshold?

> While the weights and the activations are quantized to low precision, the gradients and the optimizer states are stored in high precision to ensure training stability and accuracy. Following the previous work [ LSL+21 ], we maintain a latent weight in a high-precision format for the learnable parameters to accumulate the parameter updates. The latent weights are binarized on the fly during the forward pass and never used for the inference process.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#366

Post by barbarr »

bilsbie wrote: post #238

This really just sounds absurd. How can ternary possibly encode enough information? Anyone willing to explain it like I’m a Django developer who watched half a karpathy video?

The activations are still 8-bit, so a lot of complexity and nonlinearity is still expressible. Only the weights are 1.58-bit.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#367

Post by baq »

esrauch wrote: post #324
Noe2097 wrote: post #271

Earlier quoted context omitted.

There is another _shocking_ realization in this work: there are 11 types of people: those who know what binary means, those who don't, and those who say they do but actually don't. "The era of 1-bit LLMs" Representing { -1, 0, 1 } can't be done with 1-bit, I'm sorry -- and sad, please let's all get back to something vaguely sound and rigorous.

One trit but that's not a word anyone knows.

That used to be true yesterday…

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#368

Post by thomasahle »

mjcohen wrote: post #360
paul_mk1 wrote: post #281

Earlier quoted context omitted.

Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance…

IIRC, Hamming's book "Digital Filters" (1989) has a section on FFTs with only the sign of the coefficient being used. It performed surprisingly well.

You mean Fast Hadamard Transform?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#370

Post by gemeral »

creshal wrote: post #118
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

> * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. Easier said than done, of course, and very impressive that they pulled it off. > In matr…

> and blowing up parameter count to make up for it

based on (an admittedly rapid and indulgent reading of the paper), it seems like they're not increasing the parameter size. Do you mind pointing out where the blowup is occurring?

Post reply on HN