Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

241–250 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#241

Post by pdntspa »

shakow wrote: post #139
intelVISA wrote: post #70

Earlier quoted context omitted.

Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...

> Perhaps SWE is dead after all, but LLMs didn't kill it... Cheap electronics did. 32GB of RAM is maybe $150, a developer converting & maintaining your system to use mmap is $150k/year.

So let's toss management and go write good code for the principle of it, and not business bullshit calculus

Re: Llama.cpp 30B runs with only 6GB of RAM now

#242

Post by arka2147483647 »

terafo wrote: post #18
arka2147483647 wrote: post #13

What is lama? What can it do?

Read readme in repo.

From readme:

- Can it run doom

- Inference of LLaMA model in pure C/C++

- Plain C/C++ implementation without dependencies

It really does not explain itsef to the uniniated. I infer it is some kind of language model.

Why/how it differs any other impl/model, i do not know.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#243

Post by TaylorAlexander »

enlyth wrote: post #97
TaylorAlexander wrote: post #45

Great to see this advancing! I’m curious if anyone knows what the best repo is for running this stuff on an Nvidia GPU with 16GB vram. I ran the official repo with the leaked weights and the best I could run was the 7B parameter model. I’m curious if people have found ways to fit the larger models on such a system.

https://github.com/oobabooga/text-generation-webui

looks great thank you!

Re: Llama.cpp 30B runs with only 6GB of RAM now

#244

Post by pharmakom »

alanfranz wrote: post #238
sn_master wrote: post #196

Earlier quoted context omitted.

I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it". It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.

Python is the 2nd best language for everything. It doesn’t excel at anything, but anything a software can do, it can be done in Python somehow. So, a great pick when you’ve got no idea where you’re going to, when you’re prototyping, when you don’t care about performance or perfection. I agree that for large scale systems when you already know what you’re doing, Python shows its limits quite soon (and we should add th…

> Python is the 2nd best language for everything.

Huh? Why?

You can barely deploy it to Web.

it doesn't scale perfoance wise

you can't built robust abstractions

The REPL is merely OK

You can barely ship working code without containers

the syntax is hard to manipulate programmatically

Python has inertia but it's holding us back

Re: Llama.cpp 30B runs with only 6GB of RAM now

#245

Post by pharmakom »

elbear wrote: post #227
sn_master wrote: post #196

Earlier quoted context omitted.

I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it". It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.

It's not the easiest syntax? It's the easiest among most popular languages. It uses the least amount of symbols, parenthesis and braces only for values. Some people don't like the significant whitespace, but that helps readability.

> It uses the least amount of symbols, parenthesis and braces only for values.

is there any evidence that this makes it easier?

people learn python as beginners because it has a reputation for being easy for beginners

I don't see anything about the syntax that makes it inherently easier

Re: Llama.cpp 30B runs with only 6GB of RAM now

#246

Post by fy20 »

politician wrote: post #24
detrites wrote: post #7

The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? Ok I answered my own question.

Roughly: OpenAIs don’t employ enough jarts. In other words, the groups of folks working on training models don’t necessarily have access to the sort of optimization engineers that are working in other areas. When all of this leaked into the open, it caused a lot of people knowledgeable in different areas to put their own expertise to the task. Some of those efforts (mmap) pay off spectacularly. Expect industry to cop…

I'd say it's probably not a priority for them right now.

Of course it would save them some money if they could run their models on cheaper hardware, but they've raised $11B so I don't think that's much of a concern right now. Better to spend the efforts on pushing the model forward, which some of these optimisations may make harder.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#247

Post by pritambaral »

leereeves wrote: post #145
jart wrote: post #73

Earlier quoted context omitted.

I don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.

> You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. How is that possible? Is the model being compressed even more (even after converting to 4 bit) somehow? Or is most of the model unused?

mmap-ed memory pages backed by a file that aren't dirty aren't counted in an process's RSS usage, only kernel page cache. The mmap-ed regions of virtual memory does get counted in VSZ (virtual memory) but that is just virtual and can be larger than RAM+swap.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#248

Post by bugglebeetle »

kccqzy wrote: post #155

I might be missing something but I actually couldn't reproduce. I purposefully chose a computer with 16GiB RAM to run the 30B model. Performance was extremely slow, and the process was clearly not CPU-limited, unlike when it's running the 13B model. It's clearly swapping a lot.

Same here M1 MacBook Pro. Zero speed up on loading and inference.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#250

Post by 0xakhil »

jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

Some OS’s zram compress the unpinned pages instead of swapping to disk. It might be faster than fetching the pages again from disk. I wonder if this is a reason why folks see different results.
Post reply on HN