Looks like we can run OpenAI gpt3.5 with 32g ram?
Where are you going to download that from?
Llama.cpp 30B runs with only 6GB of RAM now
151–160 of 436 posts
Re: Llama.cpp 30B runs with only 6GB of RAM now
#152Earlier quoted context omitted.
According to https://mobile.twitter.com/JustineTunney/status/164190201019... you can probably use the conversion tools from the repo on Alpaca and get the same result. If you want to run larger Alpaca models on a low VRAM GPU, try FlexGen. I think https://github.com/oobabooga/text-generation-webui/ is one of the easier ways to get that going.
Yeah, or deepspeed presumably. Maybe torch.compile too. I dunno why I thought llama. cpp would support gpus. shrug
Lots of C++ programs use the GPU. It's irrelevant.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#153the README.md file of the repository is not updated to reflect these changes yet, right?
Re: Llama.cpp 30B runs with only 6GB of RAM now
#154Wow I continue being amazed by the progress being made on language models in the scope of weeks. I didn't expect optimisations to move this quickly. Only a few weeks ago we were amazed with ChatGPT knowing it would never be something to run at home, requiring $100.000 in hardware (8xA100 card).
Before ChatGPT was in beta, there were already models that fit into 2gb and smaller. They were complete shit, but they did exist.
I know but what's changing is that they aren't shit now. Not on par with GPT but getting much closer. Especially with a little massaging like Stanford has done.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#155I might be missing something but I actually couldn't reproduce. I purposefully chose a computer with 16GiB RAM to run the 30B model. Performance was extremely slow, and the process was clearly not CPU-limited, unlike when it's running the 13B model. It's clearly swapping a lot.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#156"How much RAM did you shave off last week?"
"Oh, you know, like 12-18GB"
"Haha shut the fuck up, how much RAM did you shave off last week"
"12-18GB"
"Let me tell you what - you show me your commits right now, if you shaved off 12-18GB of RAM last week I quit my job right now and come work for you"
Re: Llama.cpp 30B runs with only 6GB of RAM now
#157Is the 30B model clearly better than the 7B? I played with Pi3141/alpaca-lora-7B-ggml two days ago and it was super disappointing. In percentage between 0% = alpaca-lora-7B-ggml and 100% GPT-3.5, where would LLaMA 30B be positioned?
Check out the graph on page 3 of this PDF: https://arxiv.org/abs/2302.13971 The 33B model started beating the 7B when it had been trained on only 1/3 as much data. And then they kept training it to 40% more than the total that 7B was trained on. It's better.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#158Earlier quoted context omitted.
> Perhaps SWE is dead after all, but LLMs didn't kill it... Cheap electronics did. 32GB of RAM is maybe $150, a developer converting & maintaining your system to use mmap is $150k/year.
This still doesn't make sense. It doesn't take a full year to do optimizations like this. Maybe a month at most if you include the investigation time. And the memory usage is $150 times the number of users which is in the thousands at least.
Tragedy of the commons. If you want to do something that benefits everyone a little bit, and you can't productize it like OpenAI's $20/month subscription, then there's no rational economic reason to do it, and you have to wait for someone like me who has an irrational love of coding. It's not a lifestyle that makes you rich, but it does help you see the opportunities to fix problems that the well-resourced folks who are supposed to be solving them would never even notice; in fact, they'd probably think you're trolling them if you ever brought it up.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#159Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…
Just shows how inefficient some of the ML research code can be
Exactly.
It also shows the number of impostors in this thread and inflated titles of self-proclaimed 'seniors' who can't optimize ML code to even be on the same league as Tunney (jart), and Gerganov (ggerganov).
Not even ChatGPT or Copilot could even submit a change or in-fact completely rewrite and optimize this code like they have done.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#160Earlier quoted context omitted.
>What has everyone been doing wrong all these years So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost t…
AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering. You can't if you have one 12gb…
One thing I don't understand: If it's possible to chunk and parallelize it, is it not relatively straightforward to do these chunks sequentially on a single GPU with a roughly linear increase in runtime? Or are the parallelized computations actually interdependent and involving message-passing, making this unfeasible?