Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

291–300 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#291

Post by HarHarVeryFunny »

nynx wrote: post #22
jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

Why is it behaving sparsely? There are only dense operations, right?

From what I've read there's no evidence it's "behaving sparsely".. That was just offered as a suggestion why it might not be loading all the weights, but makes no sense in terms of the model. It's going to be using all the weights.

Another suggestion is that not all of the word/token embedding table might be used, which would be a function of the input used to test, but that would be easy enough to disprove as there would then be different memory usage for different inputs.

It seems possible the reported memory usage is lower than reality if that's how mmap/top work. In any case, a good use of mmap it seems, especially since for a multi-layer model layer weights will be used sequentially so paged load-on-demand will work relatively well even in a low memory situation.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#292

Post by terafo »

terafo wrote: post #57
faeriechangling wrote: post #54

Earlier quoted context omitted.

Did Meta ask permission from every user they trained their model on? Did all those users consent, and when I say consent I'm saying was there a meeting of minds not something buried in page 89 of a EULA, to Meta building an AI with their data? Turnabout is fair play. I don't feel the least bit sorry for Meta.

LLaMa was trained on data of Meta users, though.

I was sleepy, I meant to say that it WASN'T trained on data of Meta users.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#293

Post by jart »

somesortofsystm wrote: post #283
jart wrote: post #158

Earlier quoted context omitted.

Tragedy of the commons. If you want to do something that benefits everyone a little bit, and you can't productize it like OpenAI's $20/month subscription, then there's no rational economic reason to do it, and you have to wait for someone like me who has an irrational love of coding. It's not a lifestyle that makes you rich, but it does help you see the opportunities to fix problems that the well-resourced folks who…

>Tragedy of the commons. Tragedy of folks forgetting how to program. This mmap() "trick" isn't a trick, its a standard practice for anyone who has cut their teeth on POSIX or embedded. See also mlock()/munlock() ..

Well that's exactly the thing. They haven't. We're talking about a group of people here who live inside scientific papers and jupyter notebooks. They're able to make machines literally think, but you'd be pushing them out of their comfort zone if you stuck them in front of something like Emacs with C. Some people like GG, Jeff Dean, etc. are strong in both skill sets, but they're outliers.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#294

Post by renox »

pharmakom wrote: post #278
renox wrote: post #264

Earlier quoted context omitted.

You have good points but "the syntax is hard to manipulate programmatically"?? Maybe you haven't noticed but Lisp is now a tiny niche and most new languages aren't homoiconic either..

I don't think that proves anything. If we had "JavaLisp" in the browser instead of JavaScript then Lisp would be very popular. Besides that, Python is harder to manipulate than many non-Lisps, such as JavaScript and Go.

Python became popular without being the 'web language', the Lisps didn't.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#295

Post by alchemist1e9 »

freehorse wrote: post #213
alchemist1e9 wrote: post #206

Earlier quoted context omitted.

It’s interesting how NVMe will be even more critically important if this lazy weights loading approach works out. PCIe 5 has arrived just in time for LLM interference it seems.

Well in this case it does not have to do with SSDs, quite the opposite here a performance gain seems to happen by caching the file in RAM in the beginning.

That’s not my understanding. The entire point is the model can’t fit in RAM. mmap allows lazy loading from storage.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#296

Post by nunobrito »

rfoo wrote: post #232
oceanplexian wrote: post #205

Earlier quoted context omitted.

It’s not that the performance is the issue, it’s that it’s unmaintainable and prone to break. Exceptions aren’t handled right, dependencies are a disaster (Proprietary NVIDIA drivers+CUDA+PyTorch+ the various versions of stuff are a complete disaster) This leads to all sorts of bugs and breaking changes that are cool in an academic or hobbyist setting but a total headache on a large production system.

The stack is very volatile and unmaintainable because it doesn't need to be maintainable. Exactly why we have unmaintainable software in other domains. During the last 10 years there are ALWAYS totally new model architecture with new operations (or in case of CV new bizarre uses of Conv). By the time you get your performant perfectly maintainable masterpiece ready it's not needed anymore. The stack optimizes for flex…

Still a poor excuse. Had they written this in Java and things wouldn't be so difficult both on performance and maintainability.

Never understood why people think that indented languages are any simpler when in fact they bring all kinds of trouble for getting things done.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#297

Post by muyuu »

freehorse wrote: post #193
kccqzy wrote: post #155

I might be missing something but I actually couldn't reproduce. I purposefully chose a computer with 16GiB RAM to run the 30B model. Performance was extremely slow, and the process was clearly not CPU-limited, unlike when it's running the 13B model. It's clearly swapping a lot.

Same, performance of the quantised 30B model on my m1 16GB air is absolutely terrible. A couple of things I noticed on activity monitor: 1. "memory used" + "cached files" == 16GB (while swap is zero) 2. Disk reading is 500-600MB/s 3. it seems that every token is computed exactly _after every ~20GB read from disk_ which actually points that for calculating each token it actually re-reads the weights file again (instea…

Can you share the intermediate files? They're taking ages to process on my 16GB-RAM laptop

Re: Llama.cpp 30B runs with only 6GB of RAM now

#298

Post by pharmakom »

renox wrote: post #294
pharmakom wrote: post #278

Earlier quoted context omitted.

I don't think that proves anything. If we had "JavaLisp" in the browser instead of JavaScript then Lisp would be very popular. Besides that, Python is harder to manipulate than many non-Lisps, such as JavaScript and Go.

Python became popular without being the 'web language', the Lisps didn't.

Curly brace languages are more popular again.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#299

Post by alanfranz »

smallerfish wrote: post #286
alanfranz wrote: post #238

Earlier quoted context omitted.

Python is the 2nd best language for everything. It doesn’t excel at anything, but anything a software can do, it can be done in Python somehow. So, a great pick when you’ve got no idea where you’re going to, when you’re prototyping, when you don’t care about performance or perfection. I agree that for large scale systems when you already know what you’re doing, Python shows its limits quite soon (and we should add th…

To steal from another thread, Python is the McDonald's of languages - it's ubiquitous, it doesn't take much effort, and it's really not very good. The trope about it being the 2nd best language for everything isn't correct. It's taught in universities because it has a very short time to gratification, and the basic syntax is quite intuitive. Academics latched onto it for ML because of some excellent libraries, and it…

I think you’re focusing too much on the letter, rather than the idea.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#300

Post by a-dub »

intelVISA wrote: post #70
jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...

you'd be surprised how many professional programmers these days work exclusively in high level languages and know nothing about using operating system features to their fullest.

but to your point, until technology itself actually replaces us, deeply skilled computer people are always going to be able to squeeze more performance out of software implemented in high level languages by those who have not studied computers extensively.

Post reply on HN