Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

251–260 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#251

Post by cooljoseph »

anon373839 wrote: post #137
btbuildem wrote: post #68

Earlier quoted context omitted.

Discussion on HF [1] implies that no, conversion is not helpful. It would take training the model from scratch. 1: https://huggingface.co/papers/2402.17764

It’s a pity if realizing these gains absolutely requires full pre-training from scratch. I imagine more than a few people will at least try to find a way to repurpose the knowledge contained in existing models.

You can also have another model "mentor" a new model you are teaching to speed up training. You don't have to start from scratch with zero knowledge. This is done a lot in what are called distillations.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#252

Post by ordu »

jart wrote: post #60
aurareturn wrote: post #29

Earlier quoted context omitted.

After playing with OpenAI's GPT4 API, I'm quite convinced that LLMs would be in everything and everywhere today if inference cost is as low as loading a website and context size is 100x higher. In other words, only inference cost is holding it back from completely changing everything. So if we have a shortcut to getting something like GPT4 to run locally on a small device, watch out.

LLMs will give normal people a firmer standing in technological society. That's a good thing. But will it change everything? Not a chance. Even if LLMs did change everything, that probably would not be a good thing. Dijkstra says Muslim algebra died when it returned to the rhetoric style, and the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters…

> the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters of medieval scholasticism

I can propose an alternate view of things. Not that I'm going to argue that it is the only true statement in the world, but I think it is necessary for a thought to progress to have an alternative hypothesis.

So the proposition is: formal symbolisms can deal only with those problems that where already solved in imprecise human's languages.

To invent calculus and orbital mechanics you need first to talk for a several centuries (or thousands of years?) about what is position and velocity, you need to talk your way upto acceleration, and then you need to find a way to measure them and to define in a strict geometric terms. Ah, and infinity, it was a very counter-intuitive idea, Xenon invented some of his paradoxes specifically to point at counter-intuitiveness. When Newton came all these talks and debates did the most of work for him.

> the ability to understand the human tongue is insignificant compared to the power of math.

But the fun is: you cannot know if someone understands math if they do not understand human language too. You cannot teach math to those who cannot speak human language.

Math is a cream on top with a limited applicability. What math can say about love? I do not like to sound like Dumbledor, but really behind all we do there is an emotions motivating us. Math cannot deal with emotions, because it was built that way and because non-math talks about emotions hadn't bring a good model for emotions, which math could express in a formalized language.

> Dijkstra says

I wonder when he said it? Before AI concluded that expert-systems based on logic were acknowledged to be a failure or after that?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#253

Post by SushiHippie »

kouru225 wrote: post #247

Ok can someone catch me up to speed on LLM hardware requirements? Last I looked I needed a 20 gb vram card to run a good one. Is that not true anymore?

Not true anymore, but it also highly depends on what your definition of "a good one" is.

Many people find Mistral 7B to be excellent, around gpt-3.5 level of good.

Mistral 7B normally requires like 20gb VRAM, but with llama.cpp and quantization, you could even run it on your phone (albeit bad quality).

Quantization >= q4_K_M seem to provide nearly as good responses as the unquantized model, and q4_K_M only needs ~7GB of VRAM.

See the table here:

https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGU...

Using ollama you can get up and running even a bit faster than with llama.cpp directly (ollama uses llama.cpp under the hood).

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#254

Post by mikewarot »

dns_snek wrote: post #108
rafaelero wrote: post #99

Earlier quoted context omitted.

LLM's can do math as well.

Last time I checked, GPT-4 couldn't reliably add 2 numbers, never mind anything more complex.

GPT-x can't add, or subtract, or do anything else of the type... it can APPEAR to do so, because that's what it was built to do.... act like the text it's seen previously and predict what the next text would be.

If you include a large amount of properly solved math in its training text, it gets MUCH better at that kind of math.

It has a very deep set of intelligences that are alien to us, that allow it to predict and ACT LIKE us, when it comes to generating the next word. You're only seeing the output of those intelligences through a very lossy channel.

As a side note, there are structures in human language that apparently encode much more information that you might think at first glance. The fact that Word2Vec had such mathematical properties, despite it's relative simplicity, astound me to this day. Throwing a bunch of sine/cosine values on top of that to represent position in a sentence to enable LLMs is also amazing in that it works.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#255

Post by sva_ »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

I'm also curious about the potential speed gains in automatic differentiation, as there are way less branches to 'go up'. Or am I wrong here?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#256

Post by UncleOxidant »

bArray wrote: post #58
imjonse wrote: post #7

Too bad there seem to be no pretrained models to download. This is not a quantization method to apply on existing models, so having the pretrained weights is needed if one wants to test it.

+1 On this, the real proof would have been testing both models side-by-side. It seems that it may be published on GitHub [1] according to HuggingFace [2]. [1] https://github.com/microsoft/unilm/tree/master/bitnet [2] https://huggingface.co/papers/2402.17764

link #2 appears to be broken.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#257

Post by lumost »

sva_ wrote: post #255
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

I'm also curious about the potential speed gains in automatic differentiation, as there are way less branches to 'go up'. Or am I wrong here?

They actually use a relu to represent the model weights. But I'm not convinced that this can't be avoided. We do gradient boosted decision tree training without this trick.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#258

Post by arunk47 »

llm_trw wrote: post #39
anon373839 wrote: post #28

Earlier quoted context omitted.

It also means the largest models can be scaled up significantly with the same inference budget.

Depends. The only paper they cite for training: https://arxiv.org/pdf/2310.11453.pdf doesn't improve training costs much and most models are already training constrained. Not everyone has $200m to throw at training another model from scratch.

Is there any scope for indie builders?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#259

Post by esha_manideep »

These models will are compatible with llama.cpp out of the box, we (GigaML - https://gigaml.com) are planning to train a small model (3-4B, 1-bit, opensource) with the latest stack-v2 dataset released today. Let me know if anyone is interested in collaborating with us.
Post reply on HN