Fantastic article - I would love to see an analysis comparing larger quantized models to smaller unquantized models. e.g. is a 14b quantized model better than a 7b unquantized model?
How Is LLaMa.cpp Possible?
201–210 of 238 posts
Re: How Is LLaMa.cpp Possible?
#202Earlier quoted context omitted.
> some specific thing you might want to do with the model. I think this right here is the answer to measuring and comparing model performance. Instead of trying to compare models holistically, we should be comparing them for specific problem sets and use cases... the same as we compare humans against one another. Using people as an example, a hiring manager doesn't compare 2 people holistically, they compare 2 people…
You could have two models answer 100 questions the same way, and differ on the 101st. They’re unpredictable by nature - if we could accurately predict them we’d just use the predictions instead.
Well the same is true for people, and yet hiring managers still available valuate for specific tasks.
Re: How Is LLaMa.cpp Possible?
#203Earlier quoted context omitted.
you could also get HGX from any of the vendors.
Wtf does HGX mean? God enough with the acronyms people. Please take an extra ten seconds to speak in proper human language! You could save on the worlds carbon footprint by reducing the number of times humans have to search for “what is NVIDIA hgx?” or is it “what is AMD HGX” and then subsequently visiting the websites to see if that’s right or not.
What does Wft mean? God enough with the acronyms people.
/s
Re: How Is LLaMa.cpp Possible?
#204Earlier quoted context omitted.
I’ve never been a regular Twitter user, and don't really enjoy the platform, but this comment of yours is either abusing the word “shameful” or betraying a major lack of understanding that you can’t expect other people to care deeply about the things you care deeply about. There’s a lot of shameful things in this world, GP using twitter isn’t one of them. Not even a “little.”
It's betraying the word shameful in that it's an utter understatement. If people don't care about supporting companies that enable and spread far-right content and groups, it's they who are a problem. To say nothing of the pure disregard of the human right to privacy (and with AI now IP as well) that is forced on the the rest of the world by the dominance of the US market.
Apologies, but I genuinely find the intensity of this comment delusional. (1) Doesn't this perspective apply to any social media platform, for example Facebook Groups. And (2) RE: pure disregard of the human right to privacy: doesn't this implicate pretty much all popular digital platforms?
Re: How Is LLaMa.cpp Possible?
#205Earlier quoted context omitted.
ROCm works now! I just set it up tonight on a 6900xt with 16gb vram running wayland at the same time. The trick was using the opencl-amd package (somehow rocm packages don't depend on opencl, but llama does, idk). I'm astonished at the results I can get from the q6_K models.
Can you please share more info on this? I have a 6900xt "gathering dust" in a proxmox server - would like to try to do a passthrough to a vm and use it. Thank you in advance!
Same boat.
I’d love to host something local but have been so overwhelmed by the rapid progress and every time I start looking I find a guide that inevitably has a “then plug in your OpenAI api key…” step which is a hard NOPE for me.
I have a few decent gpus but I’ve got no idea where to start…
Re: How Is LLaMa.cpp Possible?
#206In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…
Any use case for using the 7B model over the 13B, quantized?
Inference speed. Sometimes 7B is good enough for the task at hand, and using 13B just makes you wait longer.
Re: How Is LLaMa.cpp Possible?
#207Earlier quoted context omitted.
What’s perplexity?
Perplexity is a measure of how certain the model is of the next token. It's calculated by looking at the probabilities that the model calculates for the next token in a stream. If there are several choices for the next token with similar probabilities, that's telling you that the model is having a hard time telling what the right answer should be: the model is more perplexed, perplexity is higher. If there's a single…
Reading this could make people believe it is computed from the probability distribution of the model alone.
To be clearer, it is the exponent of the average negative log probability that the model gives to the real tokens of a sample text[0]. Roughly, it relates to how strongly the model can predict the sample text. A perfect model would have perplexity one; a random model has a perplexity equal to the number of possible tokens; the worst model has infinite perplexity.
[0]: https://github.com/pytorch/torcheval/blob/3faf19c060b8a7c074...
Re: How Is LLaMa.cpp Possible?
#208Given the massive imbalance in the memory bandwidth bottleneck, I wonder why specialized hardware is the way it is. Is there some use case in which processing is the bottleneck, or at least it's more even? Are we expecting some software paradigm shift which will change the balance? Why couldn't they just make a cheaper, more rounded card which isn't heavily underutilized because of a large bottleneck?
IMHO the current bottleneck is driven by the partial overlap between the gaming GPU market and machine learning needs. The expensive things that are needed by both (i.e. parallel calculations) are included in all cards, but expensive things that aren't needed by games (i.e. memory bandwidth) become a bottleneck unless you use expensive ML-specific niche hardware instead of consumer (gamer) GPUs.
Re: How Is LLaMa.cpp Possible?
#209Great article. Don't see content like this anywhere else outside of HN.
Mastodon is a good way to get at this content in unfiltered form. Take a look at the raw feed of the sigmoid.social instance or follow the #LLM hashtag.
If you want it more predigested and summarized I can recommend the AI Explained channel on youtube.
Re: How Is LLaMa.cpp Possible?
#210It is useful to mention running inference on modern cpus that have AVX2 is not that bad. Sure it is slower than on the gpu, but you get the benefit of having a single long continuous region of ram. But there is one huge problem why this is not that popular on x86_64. Having to run in fp32. As far as I know our most common ml libraries (pytorch, tf, onnx etc) do not have an option to quantize to 4 bits and they don't…