Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

401–410 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#401

Post by turmeric_root »

MuffinFlavored wrote: post #373
mlgoatherder wrote: post #37

Earlier quoted context omitted.

I've done some experiments here with Llama 13B, in my subjective experience the original fp16 model is significantly better (particularly on coding tasks). There are a bunch of synthetic benchmarks such a wikitext2 PPL and all the whiz bang quantization schemes seem to score well but subjectively something is missing. I've been able to compare 4 bit GPTQ, naive int8, LLM.int8, fp16, and fp32. LLM.int8 does impressive…

> Llama 13B > Llama.cpp 30B > LLaMA-65B the "number B" stands for "number of billions" of parameters... trained on? like you take 65 billion words (from paragraphs / sentences from like, Wikipedia pages or whatever) and "train" the LLM. is that the metric? why aren't "more parameters" (higher B) always better? aka return better results how many "B" parameters is ChatGPT on GPT3.5 vs GPT4? GPT3: 175b GPT3.5: ? GPT4: ?…

> the "number B" stands for "number of billions" of parameters... trained on?

No, it's just the size of the network (i.e. number of learnable parameters). The 13/30/65B models were each trained on ~1.4 trillion tokens of training data (each token is around half a word).

Re: Llama.cpp 30B runs with only 6GB of RAM now

#402

Post by somesortofsystm »

jart wrote: post #306
somesortofsystm wrote: post #281

Earlier quoted context omitted.

> such clever use of mmap Just wanna say, that this use of mmap() is cleverly used in this context, but should be acknowledged as a widely accepted industry standard practice for getting higher performance, particularly in embedded applications but also in performance-oriented apps such as digital audio workstations, video editing systems, and so on.

Just because mmap() is commonly used doesn't mean it's commonly understood. Yes, it powers just about everything important in terms of the skeletons of our local systems. So why has the thought of using it occurred to so few people until now? Almost a whole generation has passed since things like mmap() were relegated to "the work's been done!" category of computing. People moved on to caring about things like My Bro…

The Cloud has been with us since the birth of computing. What is happening is, the computing industry goes through waves of attrition, whereby the schools push everyone up the Brand New Stack, while industry, frustrated with generations of programmers who can't program, just Builds Another Stack.

Repeat, ad infinitum. In the cracks you'll find people re-learning things they should've known, if only they weren't slagging off the grey beards .. or, even worse .. as grey beards not paying attention to the discoveries of youth.

>Most people don't know about it. The ones who do, are reluctant to use it.

Not so sure about this. The reluctance is emotional, its not technical. Nobody is killing POSIX under all of this - it is deployed. Therefore, learn it.

>so any thought devs may have had previously about using mmap() would have certainly triggered fears w.r.t. WIN32

Does not compute. Own up, you're an AI.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#403

Post by w4ffl35 »

DoctorOetker wrote: post #379
w4ffl35 wrote: post #371

Earlier quoted context omitted.

I made an app for running t5 locally - compiled version allows you to run without installing anything. https://capsizegames.itch.io/chat-ai https://github.com/Capsize-Games/chatai

interesting, what are the hardware requirements? does it happen to run on CPU on a server with 96GB RAM?

the compiled app is meant for people to install and use with their GPU and runs on as low as a GTX 1080. I haven't tested against CPU only builds.

You can take a look at the source code and see if it would be useful to you.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#404

Post by [deleted] »

shock-value wrote: post #358
jart wrote: post #325

Earlier quoted context omitted.

Please read the original link to the pull request, where I stated my change offered a 2x improvement in memory usage. You actually are able to load models 2x larger without compromising system stability, because pages are no longer being copied. That's because you previously needed 40gb of RAM to load a 20GB model, in order to ensure your file cache wasn't destroyed and need to reread from disk the next time. Now you…

It's obviously a productive change and kudos for taking it on, but much of the enthusiasm being generated here was driven by the entirely unanticipated prospect of running a model at full speed using less memory than the model's own footprint, and by the notion that inference with a dense model somehow behaved in a sparse manner at runtime. Best to be a bit more grounded here, particularly with regard to claims that…

[deleted]

Re: Llama.cpp 30B runs with only 6GB of RAM now

#405

Post by Szpadel »

lostmsu wrote: post #316
PragmaticPulp wrote: post #218

Earlier quoted context omitted.

Maybe not so fast. Other users are reporting that it’s not actually running properly in environments with limited RAM. The reduced memory usage might be more of a reporting misunderstanding, not an actual reduction in memory usage.

It will run, just will have to reread the model for every new token.

with nvme gen 4 ssds this might not be that huge of an issue, and for sure much cheaper than investing in ram

Re: Llama.cpp 30B runs with only 6GB of RAM now

#406

Post by wnoise »

miraculixx wrote: post #396
MontyCarloHall wrote: post #125

Earlier quoted context omitted.

>how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions Probably not all that much. All of the Python numeric computing frameworks (Numpy, PyTorch, TensorFlow, etc.) are basically just wrappers for lower level C++/C/Fortran code. Unless you’re doing something boneheaded and converting framework-native tensors to Python objects, passing tens…

What a bad take! Python is not the cause of dependency hell. Deep dependency trees are. The only way to deal with this is to use seperate environments and to carefully specify the exact requirements. Those who claim some language would be a magical fix clearly lack experience in multiple languages.

It's true nothing forces or forbids this, but some languages/toolings/communities/ecosystems encourage that more than others though.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#407

Post by heap_perms »

buzzert wrote: post #166
kccqzy wrote: post #132

Earlier quoted context omitted.

There are two distinct kinds of jobs: ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay; it's not their job to produce clean code. They string together libraries in Python and do a lot of experimentation and analysis. When they produced something ready to be productionized, software engineers then come in and optimize things. This is…

Funny enough, this tracks the early history of Google as well. It was originally written by Larry Page and Sergey Brin (both grad students at the time) in Python, then Sanjay Ghemawat rewrote the whole thing in C++.

That sounds very interesting. Anyone know where I can find more on this story?

Re: Llama.cpp 30B runs with only 6GB of RAM now

#408

Post by thomastjeffery »

actually_a_dog wrote: post #134
thomastjeffery wrote: post #113

Earlier quoted context omitted.

Is there any measure, not of size or token amount, but of diversity in the content of the text? Did that metric meaningfully change when the amount of required memory dropped? If the amount of diversity is lowered, I would expect that to lower the amount of patterns to be modeled from the text. If that is the case, then the resulting model size itself would be lowered, during and after training.

By "diversity," do you mean something like "entropy?" Like maybe H_s(x) := -\sum_{x \in X_s} p(x) log(p(x)) where X_s := all s-grams from the training set? That seems like it would eventually become hard to impossible to actually compute. Even if you could what would it tell you? Or, wait... are you referring to running such an analysis on the output of the model? Yeah, that might prove interesting....

I'm really just speculating here.

Because the text we write is not evenly distributed random noise, what we encode into it (by writing) is entropy.

Because LLMs model text with inference, they model all of the entropy that is present.

That would mean that the resulting size would be a measure of entropy (sum of patterns) divided by repetition (recurring patterns). In this count, I would consider each unique token alone an instance of the identity pattern.

So to answer both questions: yes.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#409

Post by lostmsu »

Szpadel wrote: post #405
lostmsu wrote: post #316

Earlier quoted context omitted.

It will run, just will have to reread the model for every new token.

with nvme gen 4 ssds this might not be that huge of an issue, and for sure much cheaper than investing in ram

I don't believe the consumer ones actually have sustained sequential read speed to saturate Gen 4.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#410

Post by nequo »

alanfranz wrote: post #336
pharmakom wrote: post #288

Earlier quoted context omitted.

Clojure JavaScript Typescript OCaml Haskell F#

Any JVM language or .NET language will take more to interface with native libraries, it’s not the same. Ocaml is very niche, I feel it’s an hard sell for a general purpose language. Haskell, 3x that. JS and TS, could be. But are they so much better than Python, if better at all?

> Ocaml is very niche, I feel it’s an hard sell for a general purpose language. Haskell, 3x that.

The impression about Haskell’s nicheness compared with OCaml prevails. But Haskell has a larger userbase and a larger library ecosystem than OCaml.

Post reply on HN