[flagged]
Llama.cpp 30B runs with only 6GB of RAM now
221–230 of 436 posts
Re: Llama.cpp 30B runs with only 6GB of RAM now
#222[flagged]
Re: Llama.cpp 30B runs with only 6GB of RAM now
#223Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…
Before, it had to read the 16GB into a buffer, which itself could be written to disk if the system needs to page out.
That is a big reason startup time is fast.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#224Earlier quoted context omitted.
>> I'm convinced entire power plants could be retired if the world stopped using python unfortunately. On the other hand, many business and professionals wouldn't exist :)
I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it". It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.
Couldn't you level the same argument against eg C++?
Re: Llama.cpp 30B runs with only 6GB of RAM now
#225Earlier quoted context omitted.
I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it". It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.
There were plenty of other languages competing with python for the same niche such as perl, ruby, js, php etc... Python is superior to all of those just for syntax alone, it is easier and cleaner to both read and write.
That might be true, but it seems to generally fall under the category of 'relevant 15+ years ago', doesn't it?
Re: Llama.cpp 30B runs with only 6GB of RAM now
#226Does that also mean 6GB VRAM? And does that include Alpaca models like this? https://huggingface.co/elinas/alpaca-30b-lora-int4
According to https://mobile.twitter.com/JustineTunney/status/164190201019... you can probably use the conversion tools from the repo on Alpaca and get the same result. If you want to run larger Alpaca models on a low VRAM GPU, try FlexGen. I think https://github.com/oobabooga/text-generation-webui/ is one of the easier ways to get that going.
Late edit: Deep Speed, not FlexGen. I don't know if FG could work, but that repo only supports it for Opt models.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#227Earlier quoted context omitted.
>> I'm convinced entire power plants could be retired if the world stopped using python unfortunately. On the other hand, many business and professionals wouldn't exist :)
I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it". It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.
It's not the easiest syntax?
It's the easiest among most popular languages. It uses the least amount of symbols, parenthesis and braces only for values.
Some people don't like the significant whitespace, but that helps readability.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#228Earlier quoted context omitted.
There were plenty of other languages competing with python for the same niche such as perl, ruby, js, php etc... Python is superior to all of those just for syntax alone, it is easier and cleaner to both read and write.
That might be true, but it seems to generally fall under the category of 'relevant 15+ years ago', doesn't it?
How do you qualify relevancy? Your own personal bubble and bias? Adoption and usage?
Pull requests and stars on github? That might be a start.
https://madnight.github.io/githut/#/pull_requests/2022/4 https://madnight.github.io/githut/#/stars/2022/4
Though you may say but but alltheprivaterepos! Then I challenge you to back up what you mean by relevance and prove python is a category of relevant 15+ years ago.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#229Re: Llama.cpp 30B runs with only 6GB of RAM now
#230Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…
Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...
Sigh.
It's not like the zero copy buzzword is going to help you during training, all your weights have to stay on GPU, you are going to sample your training data randomly and your data is on a networked storage anyway, so mmap HURTS. You'd better just O_DIRECT.
Similarly, as long as you run your inference on GPU it's not like you can mmap... And I have indeed worked on inference runtimes for mobile devices and on the rare cases we need to use CPU only (hey, your phone also have GPUs since forever) at $PREVIOUS_JOB we did have a mmap-able model format, it also helps in TEE/SGX/whatever enclave tech. Oh, and there are no Python at all.
The recent development of ggml is interesting as it catches a moment that "big ML shop infra" guys don't care: running models on Apple Silicon. M1/M2s are expensive enough that we don't consider deploying them instead of those 1000000000 bizarre accelerators in production, yet everyone on HN seems to have one and hey it's fast enough for LMs. They are rather unique as they are CPU+high bandwidth RAM+accelerators with totally shared RAM with CPU, instead of some GPU shit.
tldr it's not like "big ML shop infra" guys are stupid and leaves performance on table. They just don't run their production workload on MacBooks. That's where the community shine right?