Still can't run it, thanks AMD ROCm.
CPU inference runs perfectly fine, though.
Llama.cpp 30B runs with only 6GB of RAM now
171–180 of 436 posts
Re: Llama.cpp 30B runs with only 6GB of RAM now
#172Earlier quoted context omitted.
>What has everyone been doing wrong all these years So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost t…
>Unfortunately we don't really have a way for the FOSS community to pool together that much money to buy compute from cloud providers. How so? Why couldn't we just start a gofundme/kickstarter to fund the training of an open-source model?
who will be entrusted to benefit from this? recall that OpenAI did begin as an open source project. and that they chose to go the capitalist route, despite initially explicitly stating on their site they were a non-profit
Re: Llama.cpp 30B runs with only 6GB of RAM now
#173Earlier quoted context omitted.
~Whats a jart?~ Ah I see https://news.ycombinator.com/user?id=jart
In March 2014, Tunney petitioned the US government on We the People to hold a referendum asking for support to retire all government employees with full pensions, transfer administrative authority to the technology industry, and appoint the executive chairman of Google Eric Schmidt as CEO of America https://en.m.wikipedia.org/wiki/Justine_Tunney
what's your point? also interestingly JART is in the thread here, so they might have read your comment :)
Re: Llama.cpp 30B runs with only 6GB of RAM now
#174Where do you download the tokenizer.model that is needed to convert the GPT4ALL model to the appropriate format?
Re: Llama.cpp 30B runs with only 6GB of RAM now
#175Earlier quoted context omitted.
> But we don't have a compelling enough theory yet to explain the RAM usage miracle. My guess would be that the model is faulted into memory lazily page by page (4K or 16K chunks) as the model is used, so only the actual parts that are needed are loaded. The kernel also removes old pages from the page cache to make room for new ones, and especially so if the computer is using a lot of its RAM. As with all performance…
I don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.
> htop still reports only like 4GB of RAM is in use
I think that's just an accounting thing. Many UNIX variants will not "charge" read only memory mapped pages to a process, because they could be shared among many processes and evicted at will.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#176Earlier quoted context omitted.
How so?
There are two distinct kinds of jobs: ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay; it's not their job to produce clean code. They string together libraries in Python and do a lot of experimentation and analysis. When they produced something ready to be productionized, software engineers then come in and optimize things. This is…
I meant "how is the Python mess getting cleaned up," rather.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#177Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…
Hey, I saw your thoughtful comment before you deleted it. I just wanted to apologize — I had no idea this was a de facto Show HN, and certainly didn’t mean to make it about something other than this project. The only reason I posted it is because Facebook had been DMCAing a few repos, and I wanted to reassure everyone that they can hack freely without worry. That’s all. I’m really sorry if I overshadowed your moment…
great apology :)
Re: Llama.cpp 30B runs with only 6GB of RAM now
#178The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? Ok I answered my own question.
>What has everyone been doing wrong all these years So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost t…
> Exact training code isn't always available, and OpenAI has even gone so far as to refuse to say anything about GPT-4's architecture or training set to prevent open replication.
this is why i think the patent and copyright system is a failure. The idea that having laws protecting information like this would advance the progress of science.
It doesn't, because look how an illegally leaked model gets much more advances in shorter time. The laws protecting IP merely gives a moat to incumbents.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#179Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…
Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...
This doesn't even seem that clever, just regular ol' use of mmap where there was none before. Wonder what other performance is being left on the floor. I'm convinced entire power plants could be retired if the world stopped using python unfortunately.
Re: Llama.cpp 30B runs with only 6GB of RAM now
#180Earlier quoted context omitted.
There are two distinct kinds of jobs: ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay; it's not their job to produce clean code. They string together libraries in Python and do a lot of experimentation and analysis. When they produced something ready to be productionized, software engineers then come in and optimize things. This is…
Not sure if you came up with "Make it work, then make it right, then make it fast." but I just screenshotted it and made it the mantra for my current project...which is by far more complicated than anything I have done as a side project. I am struggling with the desire to go "make it right" as I work on shipping the deployable prototype (right now running on cloud services I control)...thanks for this
https://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast
> This formulation of this statement has been attributed to [KentBeck][0]; it has existed as part of the [UnixWay][1] for a long time.