[dead]
Llama.cpp 30B runs with only 6GB of RAM now
141–150 of 436 posts
Re: Llama.cpp 30B runs with only 6GB of RAM now
#142Earlier quoted context omitted.
>What has everyone been doing wrong all these years So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost t…
> we don't really have a way for the FOSS community to pool together that much money There must be open source projects with enough money to pool into such a project. I wonder whether wikimedia or apache are considering anything.
Maybe we can repurpose the SETI@home infrastructure :)
Re: Llama.cpp 30B runs with only 6GB of RAM now
#143Earlier quoted context omitted.
> But we don't have a compelling enough theory yet to explain the RAM usage miracle. My guess would be that the model is faulted into memory lazily page by page (4K or 16K chunks) as the model is used, so only the actual parts that are needed are loaded. The kernel also removes old pages from the page cache to make room for new ones, and especially so if the computer is using a lot of its RAM. As with all performance…
I don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.
Disk accesses should not lie. If only 6GiB are read from the disk, then I believe either the model is indeed sparse in its computation, or there may be a bug somewhere.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#144My guess is that there was an error during quantization that resulted in a large amount of the weights not properly being used. A potential test would be to compare the number of page faults between quantized an unquantized model and confirm they are roughly the same proportionally. This could also explain how e.g. gpt4all seem to notice better performance on unquantized weights when there really shouldn't be.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#145Earlier quoted context omitted.
> But we don't have a compelling enough theory yet to explain the RAM usage miracle. My guess would be that the model is faulted into memory lazily page by page (4K or 16K chunks) as the model is used, so only the actual parts that are needed are loaded. The kernel also removes old pages from the page cache to make room for new ones, and especially so if the computer is using a lot of its RAM. As with all performance…
I don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.
> You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use.
How is that possible? Is the model being compressed even more (even after converting to 4 bit) somehow? Or is most of the model unused?
Re: Llama.cpp 30B runs with only 6GB of RAM now
#146Earlier quoted context omitted.
Is the title misleading here ? 30B quantized requires 19.5 GB, not 6GB; Otherwise severe swapping to disk model original size quantized size (4-bit) 7B 13 GB 3.9 GB 13B 24 GB 7.8 GB 30B 60 GB 19.5 GB 65B 120 GB 38.5 GB
That's the size on disk, my man. When you quantize it to a smaller float size you lose precision on the weights and so the model is smaller. Then here they `mmap` the file and it only needs 6 GiB of RAM!
[deleted]
Re: Llama.cpp 30B runs with only 6GB of RAM now
#147This seems suspiciously like a bug (either in inference or in mmap reporting), as these models are not sparse enough for the savings to come from anywhere viable.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#148The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? Ok I answered my own question.
>What has everyone been doing wrong all these years So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost t…
>Unfortunately we don't really have a way for the FOSS community to pool together that much money to buy compute from cloud providers.
How so? Why couldn't we just start a gofundme/kickstarter to fund the training of an open-source model?
Re: Llama.cpp 30B runs with only 6GB of RAM now
#149Still can't run it, thanks AMD ROCm.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#150Earlier quoted context omitted.
Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...
> Perhaps SWE is dead after all, but LLMs didn't kill it... Cheap electronics did. 32GB of RAM is maybe $150, a developer converting & maintaining your system to use mmap is $150k/year.
This still doesn't make sense. It doesn't take a full year to do optimizations like this. Maybe a month at most if you include the investigation time. And the memory usage is $150 times the number of users which is in the thousands at least.