Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Is LLaMa.cpp Possible?

finbarr.ca

Post reply on HN
Print view View original

111–120 of 238 posts

Re: How Is LLaMa.cpp Possible?

#111

Post by brucethemoose2 »

Animats wrote: post #20

Is this single-thread? Or are they putting all available CPUs on the problem?

Its complicated.

If you dont have a GPU, prompt ingestion is totally threaded. The more cores the better.

For generating tokens, more cores helps to a point, but then:

- You start saturating the memory bus, and performance plateaus.

- There is some overhead from the threading implementation, and too many threads hurts performance.

The ideal number of threads varys per CPU. For instance, using hyperthreaded cores or Apple/Intel e cores typically hurts performance... But not always. You just have to test and see.

Re: How Is LLaMa.cpp Possible?

#112

Post by brucethemoose2 »

simonw wrote: post #31
_kava wrote: post #8

In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…

I was hoping that link would answer the question that's been bugging me for months: what are the penalties that you pay for using a quantized model? Sadly it didn't. It talked about "perplexities" and showed some floating point numbers. I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."

It makes the model dumber.

That seems simplistic, but its really simple as that. Naive 3 bit quantization will turn llama 7B into blubbering nonsense.

But llama.cpp quantization is good! I recommend checking out the graphs ikawrakow made for their K-quants implementation:

https://github.com/ggerganov/llama.cpp/pull/1684

Basically, the more you quantize with K-quant, the dumber the model gets. 2 bit llama 13B quant, for instance, is about as dumb as 7B F16, but the dropoff is not nearly as severe from 3-6 bits.

Re: How Is LLaMa.cpp Possible?

#113

Post by zarzavat »

ggm wrote: post #93
zarzavat wrote: post #66

Earlier quoted context omitted.

That is not what AGI means. AGI = Artificial General Intelligence. 1. Artificial = we made it 2. General = it can solve problems in any field 3. Intelligence = the ability to solve problems A chess engine is a very strong Artificial Intelligence . But it’s not very General , it can only evaluate chess positions. GPT-4 is very General , you can ask it about any question and get a somewhat reasonable answer. But it’s n…

We don't have a good computer model of dolphin intelligence and the llms are not even remotely close to consciousness or dolphins, dogs, parrots on the intelligence Front.

Consciousness ≠ intelligence.

Consciousness: being “awake” and perceiving the world.

Intelligence: solving problems, finding the truth.

Consciousness is perceiving the world, whereas intelligence is understanding it.

Re: How Is LLaMa.cpp Possible?

#114

Post by brucethemoose2 »

redox99 wrote: post #50
_kava wrote: post #8

In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…

>Still, we are looking at GPT3 level of performance on one server rack. That says something when less than a year ago, such AI was literally magic and only run on a massive datacenter. I'm not sure what you mean by this. You've always been able to run GPT3 on a single server (your typical 8xA100).

8xA100 is technically a single server, but I think OP is talking about affordable and plentiful CPU hosts, or even relatively modest single GPU instances.

DGX boxes do not grow on trees, especially these days

Re: How Is LLaMa.cpp Possible?

#117

Post by brucethemoose2 »

pedrovhb wrote: post #69

Given the massive imbalance in the memory bandwidth bottleneck, I wonder why specialized hardware is the way it is. Is there some use case in which processing is the bottleneck, or at least it's more even? Are we expecting some software paradigm shift which will change the balance? Why couldn't they just make a cheaper, more rounded card which isn't heavily underutilized because of a large bottleneck?

See: https://en.m.wikipedia.org/wiki/Random-access_memory#Memory_...

But llama.cpp, and llms in general, are very atypically memory bound, even for AI workloads. Other models/backends will typically make more use of compute and cache.

For llms specifically, cloud hosts will batch requests to make better use of GPUs.

But you are not far off. There is a proposition to pipe chips with very fast memory (aka no external memory at all) together: https://www.nextplatform.com/2023/07/12/microsofts-chiplet-c...

Re: How Is LLaMa.cpp Possible?

#118

Post by ddingus »

croes wrote: post #71
circuit10 wrote: post #32

Earlier quoted context omitted.

By most measures you could think of for intelligence languages models are improving, so I don’t see why you think this wouldn’t lead to something at least almost human-level if you scaled it up enough Of course there could be some wall somewhere but I don’t see why there would be

Because you need more training data for better results and they are running out of new training data.

I don't think so.

While it may be true that new data is coming in at a trickle these days, due to things like Discord, Slack, et al. all locking conversation and context up, as well as the daily volume of chapter is small relative to what is out there now.

The fact is that training data can be used in many different ways and I bet you we see the products of that fairly quickly as those who see this same as I do reach a point where they want to show n tell and test.

Re: How Is LLaMa.cpp Possible?

#119

Post by zarzavat »

PaulDavisThe1st wrote: post #92
zarzavat wrote: post #66

Earlier quoted context omitted.

That is not what AGI means. AGI = Artificial General Intelligence. 1. Artificial = we made it 2. General = it can solve problems in any field 3. Intelligence = the ability to solve problems A chess engine is a very strong Artificial Intelligence . But it’s not very General , it can only evaluate chess positions. GPT-4 is very General , you can ask it about any question and get a somewhat reasonable answer. But it’s n…

GPT-4 solves only 1 problem: what is the most likely stream of tokens to follow what we already have. It is remarkably good at this. But there's absolutely zero reason to believe it can solve any other problem at all.

How do humans write if not by intuiting what word comes after another?

Intelligence is the ability of that next word decision procedure to determine a next word that is aligned with our human intuition and model of truth.

I believe what you’re getting at is modality, that GPT-4 only provides responses in text. You can’t ask it to drive a car, or paint like Dall-e. And that’s a fair criticism, but it’s mostly just because it would make the models too large and slow, not because we don’t know how to do it. The thing we don’t know how to do is make a model reason as well as a human, and it makes sense to try to solve that in the text domain first rather than making highly multimodal models that reason poorly in all domains.

Re: How Is LLaMa.cpp Possible?

#120

Post by ladberg »

andrewstuart wrote: post #73

>> Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU. Sure the GPU i not fast, but if memory is important.....

You're thinking about the wrong bandwidth here. The article is talking about going from the GPU's RAM GPU cores (i.e. through load/store instructions in a cuda kernel), not from CPU's RAM GPU's RAM. That kind of bandwidth is still important but usually not the bottleneck on most ML workloads.
Post reply on HN