Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

431–436 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#431

Post by roguas »

addandsubtract wrote: post #322
simonw wrote: post #184

Earlier quoted context omitted.

"Could LLM eventually replace Google" If you try to use LLMs as a Google replacement you're going to run into problems pretty quick. LLMs are better thought of as "calculators for words" - retrieval of facts is a by-product of how they are trained, but it's not their core competence at all. LLaMA at 4bit on my laptop is around 3.9GB. There's no way you could compress all of human knowledge into less than 4GB of space…

I just need an LLM that can search, retrieve, and condense information on reddit, stackoverflow, and wikipedia to a given query.

depending on the max tokens, I think you can pretty easily fine-tune a model to return answers with actions required then wrap your prompt app to react to those, paste answers and "reask/reprompt" the same question...

similar stuff is being research under "langchains" term

Re: Llama.cpp 30B runs with only 6GB of RAM now

#432

Post by amrb »

intelVISA wrote: post #70
jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...

For the life of me I could never fix torch.load, they'll say just quantization (convert) a model to 4/8bit to make it smaller but you'll get crashes when out of system memory plus no docs.. then you admit defeat by using more swapfile :S

Re: Llama.cpp 30B runs with only 6GB of RAM now

#433

Post by irthomasthomas »

jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

Authorship of this change is contested.

https://news.ycombinator.com/item?id=35431865

Re: Llama.cpp 30B runs with only 6GB of RAM now

#434

Post by alanfranz »

nequo wrote: post #410
alanfranz wrote: post #336

Earlier quoted context omitted.

Any JVM language or .NET language will take more to interface with native libraries, it’s not the same. Ocaml is very niche, I feel it’s an hard sell for a general purpose language. Haskell, 3x that. JS and TS, could be. But are they so much better than Python, if better at all?

> Ocaml is very niche, I feel it’s an hard sell for a general purpose language. Haskell, 3x that. The impression about Haskell’s nicheness compared with OCaml prevails. But Haskell has a larger userbase and a larger library ecosystem than OCaml.

A few years have passed since I last tried out both languages. Ocaml was sort of approachable, while Haskell required quite a different mindset imho, hence the “nicheness” from the general usage standpoint.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#435

Post by Robotbeat »

lostmsu wrote: post #409
Szpadel wrote: post #405

Earlier quoted context omitted.

with nvme gen 4 ssds this might not be that huge of an issue, and for sure much cheaper than investing in ram

I don't believe the consumer ones actually have sustained sequential read speed to saturate Gen 4.

Gen 5 pcie is ~4GB/s per lane, AMD Genoa chips have 128 such lanes. That means on the order of 500GB/s aggregate throughput, which is comparable to the aggregate theoretical throughput of the 12 channel DDR5 RAM of the Genoa CPUs.

In other words, with enough data interleaving between enough NVME SSDs, you should have SSD throughput of the same order of magnitude as the system RAM.

The weights are static, so it’s just reads.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#436

Post by Robotbeat »

kmeisthax wrote: post #27
detrites wrote: post #7

The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? Ok I answered my own question.

>What has everyone been doing wrong all these years So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost t…

Training via CPU isn’t that bad if fully optimized with AVX512 extensions.
Post reply on HN