Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

71–80 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#71

Post by shepardrtc »

faeriechangling wrote: post #54
cubefox wrote: post #40

Earlier quoted context omitted.

Even if using LLaMA turns out to be legal, I very much doubt it is ethical. The model got leaked while it was only intended for research purposes. Meta engineered and paid for the training of this model. It's theirs.

Did Meta ask permission from every user they trained their model on? Did all those users consent, and when I say consent I'm saying was there a meeting of minds not something buried in page 89 of a EULA, to Meta building an AI with their data? Turnabout is fair play. I don't feel the least bit sorry for Meta.

They don't ask permission when they're stealing users' data, so why should users ask permission for stealing their data?

https://www.usatoday.com/story/tech/2022/09/22/facebook-meta...

Re: Llama.cpp 30B runs with only 6GB of RAM now

#72

Post by bakkoting »

terafo wrote: post #29
bakkoting wrote: post #14

Earlier quoted context omitted.

Some results here: https://github.com/ggerganov/llama.cpp/discussions/406 tl;dr quantizing the 13B model gives up about 30% of the improvement you get from moving from 7B to 13B - so quantized 13B is still much better than unquantized 7B. Similar results for the larger models.

I wonder where such difference between llama.cpp and [1] repo comes from. F16 difference in perplexity is .3 on 7B model, which is not insignificant. ggml quirks are definitely need to be fixed. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa

I'd guess the GPTQ-for-LLaMa repo is using a larger context size. Poking around it looks like GPTQ-for-llama is specifying 2048 [1] vs the default 512 for llama.cpp [2]. You can just specify a longer size on the CLI for llama.cpp if you are OK with the extra memory.

[1] https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/934034c8e...

[2] https://github.com/ggerganov/llama.cpp/tree/3525899277d2e2bd...

Re: Llama.cpp 30B runs with only 6GB of RAM now

#73

Post by jart »

conradev wrote: post #50
jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

> But we don't have a compelling enough theory yet to explain the RAM usage miracle. My guess would be that the model is faulted into memory lazily page by page (4K or 16K chunks) as the model is used, so only the actual parts that are needed are loaded. The kernel also removes old pages from the page cache to make room for new ones, and especially so if the computer is using a lot of its RAM. As with all performance…

I don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#74

Post by dplavery92 »

terafo wrote: post #34
kmeisthax wrote: post #27

Earlier quoted context omitted.

>What has everyone been doing wrong all these years So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost t…

AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering. You can't if you have one 12gb…

Sure, but when one 12gb GPU costs ~$800 new (e.g. for the 3080 LHR), "a couple of dozens" of them is a big barrier to entry to the hobbyist, student, or freelancer. And cloud computing offers an alternative route, but, as stated, distribution introduces a new engineering task, and the month-to-month bills for the compute nodes you are using can still add up surprisingly quickly.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#75

Post by gliptic »

terafo wrote: post #29
bakkoting wrote: post #14

Earlier quoted context omitted.

Some results here: https://github.com/ggerganov/llama.cpp/discussions/406 tl;dr quantizing the 13B model gives up about 30% of the improvement you get from moving from 7B to 13B - so quantized 13B is still much better than unquantized 7B. Similar results for the larger models.

I wonder where such difference between llama.cpp and [1] repo comes from. F16 difference in perplexity is .3 on 7B model, which is not insignificant. ggml quirks are definitely need to be fixed. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa

GPTQ-for-LLaMa recently implemented some quantization tricks suggested by the GPTQ authors that improved 7B especially. Maybe llama.cpp hasn't been evaluated with those in place?

Re: Llama.cpp 30B runs with only 6GB of RAM now

#76

Post by kossTKR »

Does this mean that we can also run the 60B model on a 16GB ram computer now?

I have the M2 air and can't wait until further optimisation with the Neural Engine / multicore gpu + shared ram etc.

I find it absolutely mind boggling that GPT-3.5(4?) level quality may be within reach locally on my $1500 laptop / $800 m2 mini.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#77

Post by seydor »

cubefox wrote: post #40
sillysaurusx wrote: post #16

On the legal front, I’ve been working with counsel to draft a counterclaim to Meta’s DMCA against llama-dl. (GPT-4 is surprisingly capable, but I’m talking to a few attorneys: https://twitter.com/theshawwn/status/1641841064800600070?s=6... ) An anonymous HN user named L pledged $200k for llama-dl’s legal defense: https://twitter.com/theshawwn/status/1641804013791215619?s=6... This may not seem like much vs Meta, but…

Even if using LLaMA turns out to be legal, I very much doubt it is ethical. The model got leaked while it was only intended for research purposes. Meta engineered and paid for the training of this model. It's theirs.

It's an index of the web and our own comments, barely something they can claim ownership on , and especially to resell.

But OTOH, by preventing commercial use, they have sparked the creation of an open source ecosystem where people are building on top of it because it's fun, not because they want to build a moat to fill it with sweet VC $$$money.

It's great to see that ecosystem being built around it, and soon someone will train a fully open source model to replace Llama

Re: Llama.cpp 30B runs with only 6GB of RAM now

#78

Post by thomastjeffery »

jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

How diverse is the training corpus?

Re: Llama.cpp 30B runs with only 6GB of RAM now

#79

Post by thomastjeffery »

kossTKR wrote: post #76

Does this mean that we can also run the 60B model on a 16GB ram computer now? I have the M2 air and can't wait until further optimisation with the Neural Engine / multicore gpu + shared ram etc. I find it absolutely mind boggling that GPT-3.5(4?) level quality may be within reach locally on my $1500 laptop / $800 m2 mini.

I doubt it: text size and text pattern size don't scale linearly.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#80

Post by conradev »

jart wrote: post #73
conradev wrote: post #50

Earlier quoted context omitted.

> But we don't have a compelling enough theory yet to explain the RAM usage miracle. My guess would be that the model is faulted into memory lazily page by page (4K or 16K chunks) as the model is used, so only the actual parts that are needed are loaded. The kernel also removes old pages from the page cache to make room for new ones, and especially so if the computer is using a lot of its RAM. As with all performance…

I don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.

Very cool! Are you testing after a reboot / with an empty page cache?
Post reply on HN