Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

131–140 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#131

Post by alchemist1e9 »

jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

I’m hopeful that when especially skilled developers like you have banged on minimizing inference resources, the others and you will start looking at distributed training ideas. Probably there is a way to decentralize the training so we can all throw in our GPUs together on building the most useful models for code generation than can be free to use and relatively cheap to run inference on. If you have any thoughts on that side of the LLM space I’m sure we would all be super curious to hear them.

Thank you for the amazing work. It’s so appreciated by so many on HN like me I’m sure.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#132

Post by kccqzy »

faitswulff wrote: post #121
abujazar wrote: post #93

I love how LLMs have got the attention of proper programmers such that the Python mess is getting cleaned up.

How so?

There are two distinct kinds of jobs: ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay; it's not their job to produce clean code. They string together libraries in Python and do a lot of experimentation and analysis. When they produced something ready to be productionized, software engineers then come in and optimize things.

This is totally the right way. Make it work, then make it right, then make it fast.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#133

Post by 8note »

electricmonk wrote: post #58
sillysaurusx wrote: post #16

On the legal front, I’ve been working with counsel to draft a counterclaim to Meta’s DMCA against llama-dl. (GPT-4 is surprisingly capable, but I’m talking to a few attorneys: https://twitter.com/theshawwn/status/1641841064800600070?s=6... ) An anonymous HN user named L pledged $200k for llama-dl’s legal defense: https://twitter.com/theshawwn/status/1641804013791215619?s=6... This may not seem like much vs Meta, but…

IANYL - This is not legal advice. As you may be aware, a counter-notice that meets the statutory requirements will result in reinstatement unless Meta sues over it. So the question isn't so much whether your counter-notice covers all the potential defenses as whether Meta is willing to sue. The primary hurdle you're going to face is your argument that weights are not creative works, and not copyrightable. That argume…

I think the counter on those arguments is that LLM owners want to avoid arguing that the model is a derivative work of the training data.

If the LLM is a specific arrangement of the copyrighted works, it's very clearly a derivative work of them

Re: Llama.cpp 30B runs with only 6GB of RAM now

#134

Post by actually_a_dog »

thomastjeffery wrote: post #113
dchest wrote: post #102

Earlier quoted context omitted.

https://arxiv.org/abs/2302.13971

Is there any measure, not of size or token amount, but of diversity in the content of the text? Did that metric meaningfully change when the amount of required memory dropped? If the amount of diversity is lowered, I would expect that to lower the amount of patterns to be modeled from the text. If that is the case, then the resulting model size itself would be lowered, during and after training.

By "diversity," do you mean something like "entropy?" Like maybe

    H_s(x) := -\sum_{x \in X_s} p(x) log(p(x))
where X_s := all s-grams from the training set? That seems like it would eventually become hard to impossible to actually compute. Even if you could what would it tell you?

Or, wait... are you referring to running such an analysis on the output of the model? Yeah, that might prove interesting....

Re: Llama.cpp 30B runs with only 6GB of RAM now

#135

Post by fallous »

bee_rider wrote: post #32
politician wrote: post #24

Earlier quoted context omitted.

Roughly: OpenAIs don’t employ enough jarts. In other words, the groups of folks working on training models don’t necessarily have access to the sort of optimization engineers that are working in other areas. When all of this leaked into the open, it caused a lot of people knowledgeable in different areas to put their own expertise to the task. Some of those efforts (mmap) pay off spectacularly. Expect industry to cop…

The professional optimizes well enough to get management off their back, the hobbyist can be irrationally good.

The professional operates within prioritization tranches. Make it work, make it reliable, make it fast, make it pretty. If you're still iterating on proof-of-concept/prototyping you'll generally confine yourself to the first and/or second levels. Once you've settled on a finalized prototype you then follow the rest of the prioritization levels to achieve shippable product.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#136

Post by startupsfail »

jart wrote: post #73
conradev wrote: post #50

Earlier quoted context omitted.

> But we don't have a compelling enough theory yet to explain the RAM usage miracle. My guess would be that the model is faulted into memory lazily page by page (4K or 16K chunks) as the model is used, so only the actual parts that are needed are loaded. The kernel also removes old pages from the page cache to make room for new ones, and especially so if the computer is using a lot of its RAM. As with all performance…

I don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.

Hmm, can you try running iotop, to see how much is being red from disk? Is it 20Gb or 6Gb? Maybe the prefetch is able to fill in before page faults are happening? Or maybe you are hitting the disk cache?

Re: Llama.cpp 30B runs with only 6GB of RAM now

#137

Post by fallous »

mlgoatherder wrote: post #37
lukev wrote: post #10

Has anyone done any comprehensive analysis on exactly how much quantization affects the quality of model output? I haven't seen any more than people running it and being impressed (or not) by a few sample outputs. I would be very curious about some contrastive benchmarks between a quantized and non-quantized version of the same model.

I've done some experiments here with Llama 13B, in my subjective experience the original fp16 model is significantly better (particularly on coding tasks). There are a bunch of synthetic benchmarks such a wikitext2 PPL and all the whiz bang quantization schemes seem to score well but subjectively something is missing. I've been able to compare 4 bit GPTQ, naive int8, LLM.int8, fp16, and fp32. LLM.int8 does impressive…

Isn't that related to architecture? The most recent GPUs and tensor procs have native support for 4-bit(partially) and 8-bit int whereas older GPUs take noticeable performance hits for 8-bit vs fp16/32.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#138

Post by chaorace »

Miraste wrote: post #110
xienze wrote: post #98

Earlier quoted context omitted.

> but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? It’s several things: * Cutting-edge code, not overly concerned with optimization * Code written by scientists, who aren’t known for being the world’s greatest programmers * The obsession the research world has with using Python Not surprising that there’s a lot of…

Why does Python get so much flak for inefficiencies? It's really not that slow, and in ML the speed-sensitive parts are libraries in lower level languages anyway. Half of the optimization from this very post is in Python.

Python has the misfortune of competing against JS in this arena, which just so happens to have the most obsessively optimized JIT ever.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#139

Post by shakow »

intelVISA wrote: post #70
jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...

> Perhaps SWE is dead after all, but LLMs didn't kill it...

Cheap electronics did. 32GB of RAM is maybe $150, a developer converting & maintaining your system to use mmap is $150k/year.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#140

Post by ImprobableTruth »

jart wrote: post #73
conradev wrote: post #50

Earlier quoted context omitted.

> But we don't have a compelling enough theory yet to explain the RAM usage miracle. My guess would be that the model is faulted into memory lazily page by page (4K or 16K chunks) as the model is used, so only the actual parts that are needed are loaded. The kernel also removes old pages from the page cache to make room for new ones, and especially so if the computer is using a lot of its RAM. As with all performance…

I don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.

What does it look like if used context size increases?
Post reply on HN