[flagged]
How Is LLaMa.cpp Possible?
101–110 of 238 posts
Re: How Is LLaMa.cpp Possible?
#102Earlier quoted context omitted.
I was hoping that link would answer the question that's been bugging me for months: what are the penalties that you pay for using a quantized model? Sadly it didn't. It talked about "perplexities" and showed some floating point numbers. I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."
I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ." We suck at evaluating and comparing models imo. There are metrics and evaluation task, but it's still very subjective. The closer we get to assessing human like performance, the tougher it is, because it becomes more subjective and less deterministic by the nature of the ta…
Not mathematically, at the very least. Perplexity is a translation of the best measure we have for informing us how a model is doing empirically over a test dataset (both pre and post). It is enough to be, usably, at least the final word on how different quantization methods perform.
Subjective ratings are different, but for compression things are quite well defined.
Re: How Is LLaMa.cpp Possible?
#103[flagged]
I'm feeling a sense of very wry irony will approach us soon....
Re: How Is LLaMa.cpp Possible?
#104Re: How Is LLaMa.cpp Possible?
#105This leaves a ton of stuff out.
- Token generation is serial and bandwidth bound, but prompt ingestion is not and runs in batches of 512+. Short tests are fast on pure CPU llama.cpp, but long prompting (such as with ongoing conversation) is extremely slow compared to other backends.
- Llama.cpp now has very good ~4 bit quantization that doesn't affect perplexity much. Q6_K almost has the same perplexity as FP16, but is still massively smaller.
- Batching is a big thing to ignore outside of personal deployments.
- The real magic of llama.cpp is model splitting. A small discrete GPU can completely offload prompt ingestion and part of the model inference. And it doesn't have to be an Nvidia GPU! There is no other backend that will do that so efficiently in the generative AI space.
- Hence the GPU backends (OpenCL, Metal, CUDA, soon ROCm and Vulkan) are the defacto way to run llama.cpp these days. Without them, I couldn't even run 70B on my desktop, or 33B on my (16GB RAM) laptop.
Re: How Is LLaMa.cpp Possible?
#106Earlier quoted context omitted.
> some specific thing you might want to do with the model. I think this right here is the answer to measuring and comparing model performance. Instead of trying to compare models holistically, we should be comparing them for specific problem sets and use cases... the same as we compare humans against one another. Using people as an example, a hiring manager doesn't compare 2 people holistically, they compare 2 people…
You could have two models answer 100 questions the same way, and differ on the 101st. They’re unpredictable by nature - if we could accurately predict them we’d just use the predictions instead.
(Stupid question) are models still non-deterministic if you set the temperature to zero?
Would setting the temperature to zero degrade the quality of response?
Re: How Is LLaMa.cpp Possible?
#107I’ve been working through that repo and managed the 13B dataset on a single Pi4 8gig I’ve also replicated the work in OpenMPI ( from a thread on the llama.cpp GitHub repo ) and today I managed to get the 65B dataset operational on three pi4 nodes. I’m not saying this as any achievement of mine, but as a comment on the current reality of reproducible LLM At home on anything you’ve got. It really feels like this techni…
> I’ve also replicated the work in OpenMPI...
Oh cool! How did it perform?
I wonder if this would be an exciting test for Amazon's SRD protocol which appears to be built for HPC. I'm looking for an excuse to play with it...
Re: How Is LLaMa.cpp Possible?
#108Earlier quoted context omitted.
His prediction was that one human brain's worth of computing power could be acquired for $1000 by 2029. That still seems reasonable. That's not the same as AGI or the singularity.
You have a metric for human brains worth of computing power which hasn't already been exceeded? I can't do infinite precision arithmetic or the RSA algorithm in my head, or index a billion strings into lexical sort order. But I am human, I am conscious and no visible VLSI work or algorithmic model will lead to AGI or a human equivalent computing power by 2029. Let alone for $1000.
well, you could just be hallucinating your own consciousness. By 2029 it seems not unreasonable to expect that the most sophisticated models will carry out visual and auditory interactions which could fool even the most sophisticated viewer. At that point, what really does consciousness mean? If the robot insisted to me it was conscious, how can I really say no?
Re: How Is LLaMa.cpp Possible?
#109Re: How Is LLaMa.cpp Possible?
#110>> Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU. Sure the GPU i not fast, but if memory is important.....
You can, right now, with the OpenCL backend.
And for the moment, its slower than pure CPU. Optimizing for IGPs is not trivial.
MLC's Vulkan backend is actually quite good on my AMD APU, but unfortunately it won't split the model to a dGPU.