Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Is LLaMa.cpp Possible?

finbarr.ca

Post reply on HN
Print view View original

131–140 of 238 posts

Re: How Is LLaMa.cpp Possible?

#131

Post by famouswaffles »

PaulDavisThe1st wrote: post #122
zarzavat wrote: post #119

Earlier quoted context omitted.

How do humans write if not by intuiting what word comes after another? Intelligence is the ability of that next word decision procedure to determine a next word that is aligned with our human intuition and model of truth. I believe what you’re getting at is modality , that GPT-4 only provides responses in text. You can’t ask it to drive a car, or paint like Dall-e. And that’s a fair criticism, but it’s mostly just be…

> How do humans write if not by intuiting what word comes after another? We don't know. It may turn out that we use mechanisms similar to LLMs, or it might be something entirely different. As for the rest: nobody knows how to make ChatGPT butter a piece of toast, let alone drive a car. ChatGPT does not reason about text, either.

>nobody knows how to make ChatGPT butter a piece of toast

There is plenty of research on LLMs successfully piloting robots.

It's but no means a solved problem but "Nobody knows how" is a stretch.

https://tidybot.cs.princeton.edu/ https://innermonologue.github.io/

>ChatGPT does not reason about text, either.

It does and there's plenty of output to demonstrate that.

Re: How Is LLaMa.cpp Possible?

#133

Post by hackernewds »

simonw wrote: post #31
_kava wrote: post #8

In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…

I was hoping that link would answer the question that's been bugging me for months: what are the penalties that you pay for using a quantized model? Sadly it didn't. It talked about "perplexities" and showed some floating point numbers. I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."

would an answer "there aren't much significant penalties" suffice?

Re: How Is LLaMa.cpp Possible?

#134

Post by sharkjacobs »

fouc wrote: post #132

Did anyone notice that apparently a M2 Macbook Pro is only 16x faster than a Pixel 5? Not sure that makes sense.

I’m not an expert in this but my “does that feel right wrong” sense isn’t going off

Pixel 5 is 2-3 years old. CPUs aren’t doubling in speed every 2 years, but let’s very generously say we expect current designs to be 4x faster than 2-3 year old equivalent.

Apple silicon is faster than other ARM chips, so if we imagine that’s another 2x we’re up to 8x

Common wisdom is that “real computers” are faster than phones, but the difference between the A16 and M2 is less than 2x for multi core, and much much less for single core benchmarks. Rounding up, another 2x is 16x

Maybe there are characteristics of the two devices which make this more surprising, I’d be interested to learn more.

Re: How Is LLaMa.cpp Possible?

#135

Post by smcleod »

_kava wrote: post #8

In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…

Good blog post, shame the site has no RSS feed!

Re: How Is LLaMa.cpp Possible?

#136

Post by DiggyJohnson »

notuseful wrote: post #78
redox99 wrote: post #56

Earlier quoted context omitted.

You can find content like this on Twitter if you follow the right people. In fact I read this article before it was even posted here because @karpathy tweeted about it.

[flagged]

I’ve never been a regular Twitter user, and don't really enjoy the platform, but this comment of yours is either abusing the word “shameful” or betraying a major lack of understanding that you can’t expect other people to care deeply about the things you care deeply about.

There’s a lot of shameful things in this world, GP using twitter isn’t one of them. Not even a “little.”

Re: How Is LLaMa.cpp Possible?

#137

Post by prvc »

_kava wrote: post #8

In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…

Any use case for using the 7B model over the 13B, quantized?

Re: How Is LLaMa.cpp Possible?

#138

Post by Roark66 »

It is useful to mention running inference on modern cpus that have AVX2 is not that bad. Sure it is slower than on the gpu, but you get the benefit of having a single long continuous region of ram.

But there is one huge problem why this is not that popular on x86_64. Having to run in fp32. As far as I know our most common ml libraries (pytorch, tf, onnx etc) do not have an option to quantize to 4 bits and they don't have an option to run inference at anything other than fp32 on the x86_64 cpus.

It is a huge shame. There is openvino which supports int8, but if you can't easily quantize large models without a gpu, what use is it? (For small models I suppose).

So if anyone figured out a way to quantize a transformer model to 4/8 bit and run it on the x86_64 cpu platform I'm very interested in hearing about it.

Re: How Is LLaMa.cpp Possible?

#139

Post by croes »

ddingus wrote: post #118
croes wrote: post #71

Earlier quoted context omitted.

Because you need more training data for better results and they are running out of new training data.

I don't think so. While it may be true that new data is coming in at a trickle these days, due to things like Discord, Slack, et al. all locking conversation and context up, as well as the daily volume of chapter is small relative to what is out there now. The fact is that training data can be used in many different ways and I bet you we see the products of that fairly quickly as those who see this same as I do reach…

>The fact is that training data can be used in many different ways and I bet you we see the products of that fairly quickly as those who see this same as I do reach a point where they want to show n tell and test.

Sounds like wishful thinking to overcome the limitations of LLMs.

At the same time we get more and more texts generated by LLMs so it gets harder to get actual man made texts.

Post reply on HN