And BTW this analysis holds true for many workloads. Memory bandwidth is often at the core of performance.
How Is LLaMa.cpp Possible?
171–180 of 238 posts
Re: How Is LLaMa.cpp Possible?
#172I enjoyed this article, but it seems to me that the latency numbers should have units of nanoseconds or maybe CPU cycles. I feel like the article was a bit sloppy with units. Another question that occurs to me is: why do chipmakers even bother putting so many functional units on the chip if almost all workloads are memory bound? Based of the calculations in this article, you could decrease the number of teraflops a m…
1. I think nanosecond-scale latency numbers on operations taking dozens to hundreds of ms are probably overkill? 2. Inference is only one aspect of what GPUs are used for. Many other workloads are compute-bound. That being said, given the recent rise of these kinds of open-source, pre-trained large language models, I wouldn't be surprised if future Nvidia product launches offered variants with significantly more VRAM…
Compromise on microseconds
Re: How Is LLaMa.cpp Possible?
#173Earlier quoted context omitted.
1. I think nanosecond-scale latency numbers on operations taking dozens to hundreds of ms are probably overkill? 2. Inference is only one aspect of what GPUs are used for. Many other workloads are compute-bound. That being said, given the recent rise of these kinds of open-source, pre-trained large language models, I wouldn't be surprised if future Nvidia product launches offered variants with significantly more VRAM…
Yeah, Nvidia won't do big VRAM consumer cards until AMD forces them to. They're running flat out just trying to keep up with demand for H100s at forty thousand USD each.
Or Intel! They're not making any high-end cards currently, but the A770 16GB is pretty decent if you're on a budget. The software support isn't really amazing yet, but their GPUs have some quite decent matmul acceleration.
Unfortunately support for their GPUs is not upstream in Tensorflow or Pytorch, and I don't think they're well-supported by Llama.cpp either, but Intel does look quite promising. I also believe that Intel oneMKL works on Arc GPUs these days, which in CPU land is amongst the fastest BLAS libraries out there. Also their matrix hardware is accessible using OpenCL extensions, which means that rolling a custom kernel for things related to quantization should be quite possible.
(For Tensorflow and PyTorch, you need to install a custom package called Intel Extension for $FRAMEWORK. The PyTorch one got updated to PyTorch 2.0 recently, which is promising.)
Currently rumors seem to indicate that their second generation GPUs will go from 32 to 64 Xe cores for the top model, and keep the 256 bit bus. If Intel were to double the VRAM to 32 GB as well (at least as an option, just like the A770 comes in both 8GB and 16GB variants), it'd immediately make them the crown of consumer VRAM size, which I'd wager would drive a lot of interest from the ML community.
Re: How Is LLaMa.cpp Possible?
#174Earlier quoted context omitted.
You're thinking about the wrong bandwidth here. The article is talking about going from the GPU's RAM GPU cores (i.e. through load/store instructions in a cuda kernel), not from CPU's RAM GPU's RAM. That kind of bandwidth is still important but usually not the bottleneck on most ML workloads.
I'm still confused. The author (correctly) made a distinction about the two like you said, but at the end when talking about Raspberry Pi 4 they use a number (~4GB/s of memory bandwidth) from an article [1] which I can only assume is NOT about graphics memory or its bandwidth (do Raspberry PIs even have it?). And how exactly is the bandwidth counted if I use integrated GPU (like i5 13600K)? Or pure CPU? [1] https://f…
Raspberry Pi's do have a usable GPU, but using it for computation is not a particularly well-traveled path. I think that's a shame. The pre-4 models have a different Broadcom graphics core to the 4, and it looks like you can get useful work out of both, but they are different enough that it's a rebuild between the generations.
Re: How Is LLaMa.cpp Possible?
#175Earlier quoted context omitted.
Humans are born with millions of years of evolutionary training embedded in their dna and brain. We are not born with nothing.
Yeah but chimpanzees and cats and mice have all the same million-year-old stuff that we do. The million-year-old stuff is not what makes humans interesting.
It's what makes humans possible. "Necessary but not sufficient" is the phrase that pays.
Re: How Is LLaMa.cpp Possible?
#176In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…
Could this be why people recently say they see more weird results in ChatGPT? Maybe OpenAI is trying out different quantization methods for the GPT4 model(s) to reduce resource usage of ChatGPT.
Re: How Is LLaMa.cpp Possible?
#177Earlier quoted context omitted.
What’s perplexity?
Perplexity is a measure of how certain the model is of the next token. It's calculated by looking at the probabilities that the model calculates for the next token in a stream. If there are several choices for the next token with similar probabilities, that's telling you that the model is having a hard time telling what the right answer should be: the model is more perplexed, perplexity is higher. If there's a single…
Thank you!
Re: How Is LLaMa.cpp Possible?
#178This leaves a ton of stuff out. - Token generation is serial and bandwidth bound, but prompt ingestion is not and runs in batches of 512+. Short tests are fast on pure CPU llama.cpp, but long prompting (such as with ongoing conversation) is extremely slow compared to other backends. - Llama.cpp now has very good ~4 bit quantization that doesn't affect perplexity much. Q6_K almost has the same perplexity as FP16, but…
ROCm works now! I just set it up tonight on a 6900xt with 16gb vram running wayland at the same time. The trick was using the opencl-amd package (somehow rocm packages don't depend on opencl, but llama does, idk).
I'm astonished at the results I can get from the q6_K models.
Re: How Is LLaMa.cpp Possible?
#179In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…
Could this be why people recently say they see more weird results in ChatGPT? Maybe OpenAI is trying out different quantization methods for the GPT4 model(s) to reduce resource usage of ChatGPT.
I'd be more inclined to believe that they're dropping down to gpt-3.5-turbo based on some heuristic, and that's why sometimes it gives you "dumber" responses. If you can serve 5/10 requests with 3.5 by swapping only the "easy" messages out, you've just cut your costs by nearly half (3.5 is like 5% of the cost of 4).