Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Is LLaMa.cpp Possible?

finbarr.ca

Post reply on HN
Print view View original

31–40 of 238 posts

Re: How Is LLaMa.cpp Possible?

#31

Post by simonw »

_kava wrote: post #8

In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…

I was hoping that link would answer the question that's been bugging me for months: what are the penalties that you pay for using a quantized model?

Sadly it didn't. It talked about "perplexities" and showed some floating point numbers.

I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."

Re: How Is LLaMa.cpp Possible?

#32

Post by circuit10 »

ggm wrote: post #13
gct wrote: post #3

Earlier quoted context omitted.

Basically Ray Kurzweil's argument, he's been saying $1000 worth of compute will be able to match human performance around 2029 for decades now.

The irony in your statement is immense. Yes, Kurzeweil has been saying this for decades. No it doesn't mean AGI is close. These llms do nothing to advance AGI. There is no theoretical basis to the belief in emergent intelligence from statistical language models and the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the dee…

By most measures you could think of for intelligence languages models are improving, so I don’t see why you think this wouldn’t lead to something at least almost human-level if you scaled it up enough

Of course there could be some wall somewhere but I don’t see why there would be

Re: How Is LLaMa.cpp Possible?

#33

Post by tysam_and »

not2b wrote: post #17
Waterluvian wrote: post #14

Earlier quoted context omitted.

Humans just do spicy autocomplete too.

No, a human isn't born with a set of knowledge like a freshly trained LLM, keeping the model fixed and responding to input. The analog to the model changes based on the human's experience. Just making bigger and bigger LLMs won't give you this.

Ah, but I believe you forget the implicit biases of genetic programming. Instincts in my experience are the skeleton, and in a sense the default basis functions for the structure of how we live, see, do, and learn.

Re: How Is LLaMa.cpp Possible?

#34

Post by gautamcgoel »

I enjoyed this article, but it seems to me that the latency numbers should have units of nanoseconds or maybe CPU cycles. I feel like the article was a bit sloppy with units.

Another question that occurs to me is: why do chipmakers even bother putting so many functional units on the chip if almost all workloads are memory bound? Based of the calculations in this article, you could decrease the number of teraflops a modern GPU can perform by a factor of 2 and not notice any appreciable difference in ML performance.

Re: How Is LLaMa.cpp Possible?

#36

Post by binary132 »

coolspot wrote: post #22
ggm wrote: post #13

Earlier quoted context omitted.

The irony in your statement is immense. Yes, Kurzeweil has been saying this for decades. No it doesn't mean AGI is close. These llms do nothing to advance AGI. There is no theoretical basis to the belief in emergent intelligence from statistical language models and the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the dee…

> the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the deep semantic meaning of the language presented. There's no intelligence. Can say the same about a half of population, tbh

So what you’re saying is.... we’re going to have human-level AI, and it’s going to be incredibly stupid

Re: How Is LLaMa.cpp Possible?

#37

Post by roarcher »

Waterluvian wrote: post #18
goatlover wrote: post #16

Earlier quoted context omitted.

No they don't. You're "just" doing what everyone else in the past has done with the brain/human intelligence and using the latest technology as a metaphor without realizing it.

We want to think we’re exceptional but all we can do is say “human consciousness is special” without having any way of measuring it or disproving the assertion that we’re just really fancy pattern matchers. Take any metaphor you want, it’s the same outcome: we may all be philosophical zombies.

We may "just" be neural networks that run on meat instead of silicon, but that does not mean that we're LLMs.

Re: How Is LLaMa.cpp Possible?

#39

Post by rootusrootus »

Aerroon wrote: post #21
Havoc wrote: post #2

What I find more stunning is what this implies going forward. If tech advances as it tends to do then having a 200bn model fit into consumer hardware isn't that far away. Might not be AGI but I think cliched as it is that would "change everything". If not at 200 then 400 or whatever. Doesn't matter - the direction of travel seems certain.

A 200b 4-bit quantized model could potentially fit into 128 GB of RAM. The inference would just be really slow. Ie you could technically run something like that today. I think more VRAM on GPUs isn't necessarily a technical limitation either. I think GPU manufacturers could add a lot more VRAM to their cards if they wanted to. The question is whether it would be worth the price increase.

> Ie you could technically run something like that today.

Yep, on higher end machines it should already be feasible. I can do 2.5-3 tok/sec on a 70B model quantized at 4 bit today with my MacBook Pro M2 MAX w/96GB. It's a little slower than a 30B, but the difference is less than I had guessed it would be. That's not super fast, but it's usable.

And that's on a machine that isn't designed for this workload. Over the next few years things should improve quite a bit. 200B does not seem like a reach.

Re: How Is LLaMa.cpp Possible?

#40

Post by eurekin »

Aerroon wrote: post #21
Havoc wrote: post #2

What I find more stunning is what this implies going forward. If tech advances as it tends to do then having a 200bn model fit into consumer hardware isn't that far away. Might not be AGI but I think cliched as it is that would "change everything". If not at 200 then 400 or whatever. Doesn't matter - the direction of travel seems certain.

A 200b 4-bit quantized model could potentially fit into 128 GB of RAM. The inference would just be really slow. Ie you could technically run something like that today. I think more VRAM on GPUs isn't necessarily a technical limitation either. I think GPU manufacturers could add a lot more VRAM to their cards if they wanted to. The question is whether it would be worth the price increase.

About the RAM. I doubt they wanted to do that, since basic gpu function is to render a frame in as little ms as possible. Currently VRAM is latency optimized on consumer gpus and all memory chips are an inch away from the gpu. Light only travels as far in the gigahertz realm. Thats why they started mounting vram chips on both sides of the board, cause there was no more place left on the first side.

Just checked: light travels 30cm in one nanosecond. So if the gpu is running at 4GHz it goes only 7.5 cm.

Post reply on HN