1. turn the swap off or monitor it closely 2. try to load a big model, like 65b-q4 or 30b-f16 3. observe the OOM - It's not so hard to test this.
Using a memory mapped file doesn't use swap. The memory is backed by the file that is memory mapped!
Llama.cpp 30B runs with only 6GB of RAM now
311–320 of 436 posts
Re: Llama.cpp 30B runs with only 6GB of RAM now
#312Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…
Hey, I saw your thoughtful comment before you deleted it. I just wanted to apologize — I had no idea this was a de facto Show HN, and certainly didn’t mean to make it about something other than this project. The only reason I posted it is because Facebook had been DMCAing a few repos, and I wanted to reassure everyone that they can hack freely without worry. That’s all. I’m really sorry if I overshadowed your moment…
I don't think you need to apologize for your comment. Even if I post a Show HN I expect no special treatment on my comments. That would be ridiculous and this is not even a Show HN.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#313Earlier quoted context omitted.
>What has everyone been doing wrong all these years So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost t…
$n00k of compute time is nothing, sorry. This is the kind of thing that academic institutions can give out for free…
why don't you write the check then huh
Re: Llama.cpp 30B runs with only 6GB of RAM now
#314Earlier quoted context omitted.
Can you share the intermediate files? They're taking ages to process on my 16GB-RAM laptop
Which files are you referring to exactly?
ggml-model-f16.bin and ggml-model-q4_0.bin
those are the output of convert-pth-to-ggml.py and quantize respectively
I had to cancel 30B as I needed to use the computer after some 12 hours, now I have to fix the ext4 filesystem of the drive where I was doing it, fun times for the weekend
guess I'll settle for 13B, I was using 7B but the results are pretty lousy compared to GPT4all's Lora, let alone GPT3.5-turbo or better
I'll give a shot to quantising 13B, I'm on 16GB of RAM locally
Re: Llama.cpp 30B runs with only 6GB of RAM now
#315Earlier quoted context omitted.
Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...
You can mmap from python.
In fact, you can mmap from PyTorch directly.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#316"How much RAM did you shave off last week?" "Oh, you know, like 12-18GB" "Haha shut the fuck up, how much RAM did you shave off last week" "12-18GB" "Let me tell you what - you show me your commits right now, if you shaved off 12-18GB of RAM last week I quit my job right now and come work for you" https://www.youtube.com/watch?v=TxHITqC5rxE
Maybe not so fast. Other users are reporting that it’s not actually running properly in environments with limited RAM. The reduced memory usage might be more of a reporting misunderstanding, not an actual reduction in memory usage.
It will run, just will have to reread the model for every new token.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#317Earlier quoted context omitted.
It's not the easiest syntax? It's the easiest among most popular languages. It uses the least amount of symbols, parenthesis and braces only for values. Some people don't like the significant whitespace, but that helps readability.
> It uses the least amount of symbols, parenthesis and braces only for values. is there any evidence that this makes it easier? people learn python as beginners because it has a reputation for being easy for beginners I don't see anything about the syntax that makes it inherently easier
And what symbols it has, it reuses them wisely.
The square brackets alone make it a winner. Array, list and strings indexing. Dictionary lookups. Slices and substrings. List comprehensions. The notations convenience of this alone is immense.
Built in list, string, and dicts. For the 90% of code that is not performance critical, this is a godsend. Just looking at the c++ syntax for this makes me never want to use a stl data structure for anything trivial.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#318Earlier quoted context omitted.
> Perhaps SWE is dead after all, but LLMs didn't kill it... Cheap electronics did. 32GB of RAM is maybe $150, a developer converting & maintaining your system to use mmap is $150k/year.
So let's toss management and go write good code for the principle of it, and not business bullshit calculus
Good, tell me how your company will be doing.
What people sometimes fail to understand is that code is a mean to an end, not an end in itself.
If you want to make code for itself, work on an opensource and/or personal project. If you are paid to work on something, you're paid for the something to get out, not for it to feature the best code ever.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#319Earlier quoted context omitted.
How so?
C has an almost infinite horizon for optimization. Python is good prototypes but we are beyond that stage now
99% of LLM evaluation with PyTorch was already done in C++.
These .cpp projects don't improve anything for performance. They just drop dependencies necessary for training and experimentation.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#320Earlier quoted context omitted.
C has an almost infinite horizon for optimization. Python is good prototypes but we are beyond that stage now
99% of LLM evaluation with PyTorch was already done in C++. These .cpp projects don't improve anything for performance. They just drop dependencies necessary for training and experimentation.
Optimization isn't just about speed. As you said, dropping dependencies makes it portable, embeddable, more versatile