Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

111–120 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#111

Post by stavros »

terafo wrote: post #39
bsaul wrote: post #26

how is llama performance relative to chatgpt ? is it as good as chatgpt3 or even 4 ?

It is as good as GPT-3 at most sizes. Instruct layer needs to be put on top in order for it to compete with GPT 3.5(which powers ChatGPT). It can be done with comparatively little amount of compute(couple hundred bucks worth of compute for small models, I'd assume low thousands for 65B).

That's surprising to read, given that ChatGPT (at least the first version) was much worse than text-davinci-003 at following instructions. The new version seems to be much better, though.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#113

Post by thomastjeffery »

dchest wrote: post #102
thomastjeffery wrote: post #78

Earlier quoted context omitted.

How diverse is the training corpus?

https://arxiv.org/abs/2302.13971

Is there any measure, not of size or token amount, but of diversity in the content of the text?

Did that metric meaningfully change when the amount of required memory dropped?

If the amount of diversity is lowered, I would expect that to lower the amount of patterns to be modeled from the text. If that is the case, then the resulting model size itself would be lowered, during and after training.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#114

Post by dsjoerg »

politician wrote: post #24
detrites wrote: post #7

The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? Ok I answered my own question.

Roughly: OpenAIs don’t employ enough jarts. In other words, the groups of folks working on training models don’t necessarily have access to the sort of optimization engineers that are working in other areas. When all of this leaked into the open, it caused a lot of people knowledgeable in different areas to put their own expertise to the task. Some of those efforts (mmap) pay off spectacularly. Expect industry to cop…

~Whats a jart?~

Ah I see https://news.ycombinator.com/user?id=jart

Re: Llama.cpp 30B runs with only 6GB of RAM now

#115

Post by nmca »

yodsanklai wrote: post #101

Total noob questions. 1. How does this compare with ChatGPT3 2. Does it mean we could eventually run a system such as ChatGPT3 on a computer 3. Could LLM eventually replace Google (in the sense that answers could be correct 99.9% of the time) or is the tech inherently flawed

worse, yes, yes, no

Re: Llama.cpp 30B runs with only 6GB of RAM now

#116

Post by smoldesu »

wkat4242 wrote: post #104

Wow I continue being amazed by the progress being made on language models in the scope of weeks. I didn't expect optimisations to move this quickly. Only a few weeks ago we were amazed with ChatGPT knowing it would never be something to run at home, requiring $100.000 in hardware (8xA100 card).

Before ChatGPT was in beta, there were already models that fit into 2gb and smaller. They were complete shit, but they did exist.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#117

Post by actually_a_dog »

htrp wrote: post #67
jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

Just shows how inefficient some of the ML research code can be

As a former grad student, I can tell you, that's all research code, not just ML, or even "performance-oriented" research code.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#118

Post by chatmasta »

Miraste wrote: post #110
xienze wrote: post #98

Earlier quoted context omitted.

> but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? It’s several things: * Cutting-edge code, not overly concerned with optimization * Code written by scientists, who aren’t known for being the world’s greatest programmers * The obsession the research world has with using Python Not surprising that there’s a lot of…

Why does Python get so much flak for inefficiencies? It's really not that slow, and in ML the speed-sensitive parts are libraries in lower level languages anyway. Half of the optimization from this very post is in Python.

It might not be slow in general, but it's easy to write slow code in it.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#119

Post by jart »

eternalban wrote: post #30
jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

Great work. Is the new file format described anywhere? Skimming the issue comments I have a vague sense that r/o matter was colocated somewhere for zero copy mmap or is there more to it?

That's something I'm working on presently.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#120

Post by sp332 »

dsjoerg wrote: post #114
politician wrote: post #24

Earlier quoted context omitted.

Roughly: OpenAIs don’t employ enough jarts. In other words, the groups of folks working on training models don’t necessarily have access to the sort of optimization engineers that are working in other areas. When all of this leaked into the open, it caused a lot of people knowledgeable in different areas to put their own expertise to the task. Some of those efforts (mmap) pay off spectacularly. Expect industry to cop…

~Whats a jart?~ Ah I see https://news.ycombinator.com/user?id=jart

In March 2014, Tunney petitioned the US government on We the People to hold a referendum asking for support to retire all government employees with full pensions, transfer administrative authority to the technology industry, and appoint the executive chairman of Google Eric Schmidt as CEO of America

https://en.m.wikipedia.org/wiki/Justine_Tunney

Post reply on HN