Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Is LLaMa.cpp Possible?

finbarr.ca

Post reply on HN
Print view View original

141–150 of 238 posts

Re: How Is LLaMa.cpp Possible?

#141

Post by class4behavior »

DiggyJohnson wrote: post #136
notuseful wrote: post #78

Earlier quoted context omitted.

[flagged]

I’ve never been a regular Twitter user, and don't really enjoy the platform, but this comment of yours is either abusing the word “shameful” or betraying a major lack of understanding that you can’t expect other people to care deeply about the things you care deeply about. There’s a lot of shameful things in this world, GP using twitter isn’t one of them. Not even a “little.”

It's betraying the word shameful in that it's an utter understatement.

If people don't care about supporting companies that enable and spread far-right content and groups, it's they who are a problem.

To say nothing of the pure disregard of the human right to privacy (and with AI now IP as well) that is forced on the the rest of the world by the dominance of the US market.

Re: How Is LLaMa.cpp Possible?

#142

Post by pjmlp »

wfurney wrote: post #88
jmorgan wrote: post #9

This project's been a blast to work with. While it's written in C++, it provides a C interface to compile against which makes it especially easy to extend with Go, Python and other runtimes. A few folks and I have been building a tool with it in Go for pulling & running multiple models, and serving them on a REST API: https://github.com/jmorganca/ollama In similar light, you haven't checked it out, llama.cpp also has…

I've also been experimenting with the C# implementation https://github.com/trrahul/llama2.cs

Why Parallel.For() instead of SIMD types?

I would expect a much better performance.

Re: How Is LLaMa.cpp Possible?

#143

Post by YetAnotherNick »

_kava wrote: post #8

In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…

The effect is lesser than you think. 5 bit quantization has negligible performance loss compared to 16 bits: https://github.com/ggerganov/llama.cpp/pull/1684

Re: How Is LLaMa.cpp Possible?

#144

Post by dustymcp »

simonw wrote: post #31
_kava wrote: post #8

In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…

I was hoping that link would answer the question that's been bugging me for months: what are the penalties that you pay for using a quantized model? Sadly it didn't. It talked about "perplexities" and showed some floating point numbers. I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."

it will be different for every usecase, the only way to find out is spinning one up..

Re: How Is LLaMa.cpp Possible?

#146

Post by astrange »

YetAnotherNick wrote: post #143
_kava wrote: post #8

In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…

The effect is lesser than you think. 5 bit quantization has negligible performance loss compared to 16 bits: https://github.com/ggerganov/llama.cpp/pull/1684

This paper from last month has a method for acceptable 3-bit quantization and a start at 2-bit.

https://arxiv.org/abs/2307.13304

Re: How Is LLaMa.cpp Possible?

#147

Post by astrange »

PaulDavisThe1st wrote: post #92
zarzavat wrote: post #66

Earlier quoted context omitted.

That is not what AGI means. AGI = Artificial General Intelligence. 1. Artificial = we made it 2. General = it can solve problems in any field 3. Intelligence = the ability to solve problems A chess engine is a very strong Artificial Intelligence . But it’s not very General , it can only evaluate chess positions. GPT-4 is very General , you can ask it about any question and get a somewhat reasonable answer. But it’s n…

GPT-4 solves only 1 problem: what is the most likely stream of tokens to follow what we already have. It is remarkably good at this. But there's absolutely zero reason to believe it can solve any other problem at all.

That's an interface, not an implementation.

("The most likely" out of what distribution? The model's distribution. So that just means "what the model thinks the answer to your question is".)

Re: How Is LLaMa.cpp Possible?

#148

Post by astrange »

fouc wrote: post #132

Did anyone notice that apparently a M2 Macbook Pro is only 16x faster than a Pixel 5? Not sure that makes sense.

The phone slows down as it heats up, and then slows down infinitely when the battery dies.

(With some workloads and chargers, it's possible running on a phone would drain the battery faster than it can recharge even if you plugged it in.)

Re: How Is LLaMa.cpp Possible?

#149

Post by clarionbell »

prvc wrote: post #137
_kava wrote: post #8

In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…

Any use case for using the 7B model over the 13B, quantized?

SBC

Re: How Is LLaMa.cpp Possible?

#150

Post by _nalply »

Roark66 wrote: post #138

It is useful to mention running inference on modern cpus that have AVX2 is not that bad. Sure it is slower than on the gpu, but you get the benefit of having a single long continuous region of ram. But there is one huge problem why this is not that popular on x86_64. Having to run in fp32. As far as I know our most common ml libraries (pytorch, tf, onnx etc) do not have an option to quantize to 4 bits and they don't…

Subjective experience: AVX512 helps a lot. I would have liked to read more about this. It seems that AVX512 supports fp16 in hardware and allows 32 fused multiplication-add operations per core. So I imagine on a Ryzen 9 with 12 cores you can have 384 simultaneous fused multiplication-add operations. I am not sure whether my estimation is off. Anyone know more than me?
Post reply on HN