Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

421–430 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#421

Post by gpderetta »

Noe2097 wrote: post #271
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

There is another _shocking_ realization in this work: there are 11 types of people: those who know what binary means, those who don't, and those who say they do but actually don't. "The era of 1-bit LLMs" Representing { -1, 0, 1 } can't be done with 1-bit, I'm sorry -- and sad, please let's all get back to something vaguely sound and rigorous.

There are 10 types of people, those who don't know binary, those who do and those who know ternary.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#422

Post by libertalia0 »

esha_manideep wrote: post #259

These models will are compatible with llama.cpp out of the box, we (GigaML - https://gigaml.com ) are planning to train a small model (3-4B, 1-bit, opensource) with the latest stack-v2 dataset released today. Let me know if anyone is interested in collaborating with us.

Highly interested in collaborating – got a bunch of proprietary legal data already pre-sorted and labeled for various scenarios. I've already benchmarked legal use-cases (i.e. legal speciality, a few logic-based questions, and specific document creation) with various LLMs – so would love to see what benchmarks this can produced compared to early Mistral or Llama.

Let me know what's the best way to reach out!

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#424

Post by tbalsam »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

It's not too surprising, honestly! I've poked around with similar in the past and am of a perspective that ternary is a very good thing for a lot of neural networks.

Training CIFAR-10 speedily w/ ternary weights on an fp16 interface (using fp16 buffers, and norm params unchanged): https://gist.github.com/tysam-code/a43c0fab332e50163b74141bc...

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#425

Post by kristjansson »

pclmulqdq wrote: post #292
vessenes wrote: post #232

Earlier quoted context omitted.

Update - I'm still cautious about this paper, but I had the table numbers inverted in my head while thinking about it. The paper shows better perplexity results than competing models at larger parameter sizes, so I was wrong.

I was pretty unhappy and suspicious for the same reason. Not reporting perplexity for a 70B network while reporting its efficiency means that someone did something and the result wasn't good enough to put in the paper.

One can forgive the lack of quality results for the 70B model, but apparently they trained 7B and 13B versions of their model, and don't report those either.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#426

Post by hk__2 »

Noe2097 wrote: post #271
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

There is another _shocking_ realization in this work: there are 11 types of people: those who know what binary means, those who don't, and those who say they do but actually don't. "The era of 1-bit LLMs" Representing { -1, 0, 1 } can't be done with 1-bit, I'm sorry -- and sad, please let's all get back to something vaguely sound and rigorous.

> please let's all get back to something vaguely sound and rigorous

Something rigorous would be to actually read the paper rather than stop at the first part of its title. The authors are not claiming their LLM is 1-bit.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#427

Post by bArray »

UncleOxidant wrote: post #256
bArray wrote: post #58

Earlier quoted context omitted.

+1 On this, the real proof would have been testing both models side-by-side. It seems that it may be published on GitHub [1] according to HuggingFace [2]. [1] https://github.com/microsoft/unilm/tree/master/bitnet [2] https://huggingface.co/papers/2402.17764

link #2 appears to be broken.

Tested earlier, still seems to be working fine. I can only suggest to try a VPN/alternative DNS?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#428

Post by Blackthorn »

riskable wrote: post #414
Blackthorn wrote: post #203

Is there any rigorous way to answer the question of how much information (be it entropy or some other measurement) is contained in a model's weights?

Yes, actually: That's the entire point of the paper! The concept is that the amount of information contained in a weight like 0.00006103515625 is equivalent to 0. -0.99951172 is equivalent to -1, 1.26406236 equivalent to 1, etc. That there's no practical difference when actually utilizing the model (if trained in ternary from the start). The paper posits (and provides evidence) that if you train a model using ternary…

I don't think you really answered my question. What's been done by the paper is show experimentally that networks don't have enough information to justify their weight precision, and that's really good and a very important result, but what I was asking was if there's a rigorous way to take an arbitrary network and determine its information content (either by itself, or compared to another network). Possibly that can be relative to its outputs.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#429

Post by gerash »

jdthedisciple wrote: post #398

People have been doing this 6 years ago. https://github.com/yashkant/quantized-nets https://github.com/TropComplique/trained-ternary-quantization https://github.com/buaabai/Ternary-Weights-Network I too find it very interesting. But why this sudden, renewed fuzz?

I haven't read the paper but I clearly remember 1-bit quantization from at least 5-6 years ago

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#430

Post by fl0ki »

Would there be value in distinguishing -0 and +0? If a 0 was quantized from a small negative or a small positive, it seems like retaining the sign is better than forgetting it.

The question remains whether the benefit and the simpler design are worth the loss of density.

Post reply on HN