Llama.cpp 30B runs with only 6GB of RAM now
1–10 of 436 posts
Re: Llama.cpp 30B runs with only 6GB of RAM now
#2Does that also mean 6GB VRAM?
And does that include Alpaca models like this? https://huggingface.co/elinas/alpaca-30b-lora-int4
Re: Llama.cpp 30B runs with only 6GB of RAM now
#3[deleted]
Re: Llama.cpp 30B runs with only 6GB of RAM now
#4Does that also mean 6GB VRAM? And does that include Alpaca models like this? https://huggingface.co/elinas/alpaca-30b-lora-int4
No(llama.cpp is cpu-only) and no(you need to requantize the model).
Re: Llama.cpp 30B runs with only 6GB of RAM now
#5Does that also mean 6GB VRAM? And does that include Alpaca models like this? https://huggingface.co/elinas/alpaca-30b-lora-int4
According to https://mobile.twitter.com/JustineTunney/status/164190201019... you can probably use the conversion tools from the repo on Alpaca and get the same result.
If you want to run larger Alpaca models on a low VRAM GPU, try FlexGen. I think https://github.com/oobabooga/text-generation-webui/ is one of the easier ways to get that going.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#6I wonder if Georgi or jart use GPT in their programming and design. I guess the training data was lacking for the sort of stuff they do due to their field of work especially jart.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#7The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks?
Ok I answered my own question.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#8Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome, but like all things in science a small amount of healthy skepticism is warranted.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#9The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? Ok I answered my own question.
I have predicted that LLaMA will be available on mobile phones before the end of this year. We are very close.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#10Has anyone done any comprehensive analysis on exactly how much quantization affects the quality of model output? I haven't seen any more than people running it and being impressed (or not) by a few sample outputs.
I would be very curious about some contrastive benchmarks between a quantized and non-quantized version of the same model.