Llama.cpp 30B runs with only 6GB of RAM now • Hacker News Board

Re: Llama.cpp 30B runs with only 6GB of RAM now

#321

Post by [deleted] » Sat, Apr 01, 2023, 2:29 PM UTC

leereeves wrote: post #145

jart wrote: post #73

Earlier quoted context omitted.

I don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.

> You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. How is that possible? Is the model being compressed even more (even after converting to 4 bit) somehow? Or is most of the model unused?

[deleted]

Re: Llama.cpp 30B runs with only 6GB of RAM now

#322

Post by addandsubtract » Sat, Apr 01, 2023, 2:41 PM UTC

simonw wrote: post #184

yodsanklai wrote: post #101

Total noob questions. 1. How does this compare with ChatGPT3 2. Does it mean we could eventually run a system such as ChatGPT3 on a computer 3. Could LLM eventually replace Google (in the sense that answers could be correct 99.9% of the time) or is the tech inherently flawed

"Could LLM eventually replace Google" If you try to use LLMs as a Google replacement you're going to run into problems pretty quick. LLMs are better thought of as "calculators for words" - retrieval of facts is a by-product of how they are trained, but it's not their core competence at all. LLaMA at 4bit on my laptop is around 3.9GB. There's no way you could compress all of human knowledge into less than 4GB of space…

I just need an LLM that can search, retrieve, and condense information on reddit, stackoverflow, and wikipedia to a given query.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#323

Post by DoctorOetker » Sat, Apr 01, 2023, 2:41 PM UTC

Is there a reason Llama is getting so much attention compared to say T5 11B?

Not sure how neutral or what benchmarks are used on the following link, but T5 seems to sit a lot higher on this leaderboard?

https://accubits.com/large-language-models-leaderboard/

Re: Llama.cpp 30B runs with only 6GB of RAM now

#324

Post by freehorse » Sat, Apr 01, 2023, 2:42 PM UTC

diimdeep wrote: post #280

xiphias2 wrote: post #167

Earlier quoted context omitted.

Now it's clear that there was a bug in the measurement. The author used a machine with lots of RAM, so I guess most of us are still stuck with quantized 13B. Still, the improvement hopefully translates, and I hope that 30B will run with 3 bit quantization in a few days.

Also, current SSD's achieve 7.5 GB/s+ read speed, opposed to older SSD from 2013 with 500 MB/s, so performance will drastically differ depending on your system specs in case of pulling weights from disk to RAM on demand. Also, there is $ vmmap where we can see various statistics about process memory and used swap, that are not available in top or htop.

Even with 7.5GB/s you are gonna at best achieve 2.7 seconds for a computing a token, in a hyperoptimistic scenario that you can actually achieve that speed in reading the file, which is too slow for doing much. Maybe if one could get the kernel to swap more aggressively or sth it could cut half that time or so, but it still would be quite slow.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#325

Post by jart » Sat, Apr 01, 2023, 2:45 PM UTC

shock-value wrote: post #303

rcarmo wrote: post #251

Earlier quoted context omitted.

This is nothing short of legendary. Was following the thread on Twitter and LOLed at the replies of “Praise be Jart”, but there’s something of the sublime here. Great weight wrangling judo :)

It appears that this was just a misreading of how memory usage was being reported and there was actually no improvement here. At least nothing so sensational as being able to run a larger-than-RAM model without swapping from disk on every iteration.

Please read the original link to the pull request, where I stated my change offered a 2x improvement in memory usage. You actually are able to load models 2x larger without compromising system stability, because pages are no longer being copied. That's because you previously needed 40gb of RAM to load a 20GB model, in order to ensure your file cache wasn't destroyed and need to reread from disk the next time. Now you only need 20GB to load a 20GB model.

The peculiarity here is that tools like htop were reporting the improvement as being an 8x improvement, which is interesting, because RAM use is only 2x better due to my change. The rusage.com page fault reporting was also interesting too. This is not due to sparseness. It's because htop was subtracting MAP_SHARED memory. The htop docs say on my computer that the color purple is used to display shared memory, and yellow is used to display kernel file caches. But it turned out it just uses yellow for both, even though it shouldn't, because mincore() reported that the shared memory had been loaded into the resident set size.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#326

Post by revelio » Sat, Apr 01, 2023, 2:50 PM UTC

nunobrito wrote: post #296

rfoo wrote: post #232

Earlier quoted context omitted.

The stack is very volatile and unmaintainable because it doesn't need to be maintainable. Exactly why we have unmaintainable software in other domains. During the last 10 years there are ALWAYS totally new model architecture with new operations (or in case of CV new bizarre uses of Conv). By the time you get your performant perfectly maintainable masterpiece ready it's not needed anymore. The stack optimizes for flex…

Still a poor excuse. Had they written this in Java and things wouldn't be so difficult both on performance and maintainability. Never understood why people think that indented languages are any simpler when in fact they bring all kinds of trouble for getting things done.

There's a Java ML library called Tribuo that might be worth looking at.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#327

Post by dekhn » Sat, Apr 01, 2023, 3:01 PM UTC

jart wrote: post #95

conradev wrote: post #80

Earlier quoted context omitted.

Very cool! Are you testing after a reboot / with an empty page cache?

Pretty much. I do my work on a headless workstation that I SSH into, so it's not like competing with Chrome tabs or anything like that. But I do it mostly because that's what I've always done. The point of my change is you won't have to be like me anymore. Many of the devs who contacted after using my change have been saying stuff like, "yes! I can actually run LLaMA without having to close all my apps!" and they're…

Linux has a command to drop caches at runtime (https://www.tecmint.com/clear-ram-memory-cache-buffer-and-sw...) which is VERY useful during debugging.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#328

Post by dopidopHN » Sat, Apr 01, 2023, 3:06 PM UTC

revelio wrote: post #326

nunobrito wrote: post #296

Earlier quoted context omitted.

Still a poor excuse. Had they written this in Java and things wouldn't be so difficult both on performance and maintainability. Never understood why people think that indented languages are any simpler when in fact they bring all kinds of trouble for getting things done.

There's a Java ML library called Tribuo that might be worth looking at.

Thanks, the boring aspect of Java is appealing here.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#329

Post by breck » Sat, Apr 01, 2023, 3:07 PM UTC

chii wrote: post #178

kmeisthax wrote: post #27

Earlier quoted context omitted.

>What has everyone been doing wrong all these years So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost t…

> Exact training code isn't always available, and OpenAI has even gone so far as to refuse to say anything about GPT-4's architecture or training set to prevent open replication. this is why i think the patent and copyright system is a failure. The idea that having laws protecting information like this would advance the progress of science. It doesn't, because look how an illegally leaked model gets much more advance…

> The laws protecting IP merely gives a moat to incumbents.

Yes. These laws are bad. We could fix this with a 2 line change:

    Section 1. Article I, Section 8, Clause 8 of this Constitution is hereby repealed.
    Section 2. Congress shall make no law abridging the right of the people to publish information.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#330

Post by dekhn » Sat, Apr 01, 2023, 3:09 PM UTC

muyuu wrote: post #314

freehorse wrote: post #305

Earlier quoted context omitted.

Which files are you referring to exactly?

ggml-model-f16.bin and ggml-model-q4_0.bin those are the output of convert-pth-to-ggml.py and quantize respectively I had to cancel 30B as I needed to use the computer after some 12 hours, now I have to fix the ext4 filesystem of the drive where I was doing it, fun times for the weekend guess I'll settle for 13B, I was using 7B but the results are pretty lousy compared to GPT4all's Lora, let alone GPT3.5-turbo or bet…

Yeah, the first time I ran the 30B model, it crashed my machine and I had to reinstall from scratch (linux).