Llama.cpp 30B runs with only 6GB of RAM now • Hacker News Board

Re: Llama.cpp 30B runs with only 6GB of RAM now

#301

Post by muyuu » Sat, Apr 01, 2023, 1:10 PM UTC

Dwedit wrote: post #100

> 6GB of RAM > Someone mentioning "32-bit systems" Um no, you're not mapping 6GB on RAM on a 32-bit system. The address space simply doesn't exist.

It actually happened. With paging and bigger word sizes, which were common in 32 bit systems even with hardware acceleration.

A 32bit address space only means you have 4GibiAddresses, which do not need to be pointing to single bytes. In fact the natural thing to do in a 32 bit system for a structure like this is moving 32bit words, which actually means you're addressing a 16GB space, flat. And then there's segmentation.

For instance the 286 had a 24bit address showing for 16MB in direct addressing mode and 1GB via segmentation (what back then was usually referred to by virtual memory)

The 386 had a 32 bit address width and its MMU allowed access to 64TB in virtual mode and 4GB in protected mode.This was indeed one of the reasons Linux was not made 286-compatible. Its protected mode was only 1GB and segmented rather than 4GB flat, so Linus didn't have to deal with XMS or EMS for a chip that was becoming obsolete soon anyway. But the 1GB space was there, and at the time that was plenty.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#302

Post by pablo1107 » Sat, Apr 01, 2023, 1:11 PM UTC

version_five wrote: post #254

bboygravity wrote: post #252

Earlier quoted context omitted.

No idea what a Google Collab is, but does the code come with an environment or at least a specifications of which packages and versions to use (requirements.txt)? It sounds unnecessarily weird to me that people would share Python code that simply doesn't work out at all out of the box.

Its rarely as easy as sharing a requirements.txt. There are lots of things that can still break - for examples you get weird situations where different modules require different versions of a third module. Or all the Cuda toolkit version issues thsy seem to come up with gpu stuff. When we share python, we tend to share a docker image, and even this isn't foolproof. A big problem I think is that it doesn't incentivize…

Maybe using Nix it's a better experience for creating such an environment where you depending also on system utilities.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#303

Post by shock-value » Sat, Apr 01, 2023, 1:16 PM UTC

rcarmo wrote: post #251

jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

This is nothing short of legendary. Was following the thread on Twitter and LOLed at the replies of “Praise be Jart”, but there’s something of the sublime here. Great weight wrangling judo :)

It appears that this was just a misreading of how memory usage was being reported and there was actually no improvement here. At least nothing so sensational as being able to run a larger-than-RAM model without swapping from disk on every iteration.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#304

Post by deadly_syn » Sat, Apr 01, 2023, 1:18 PM UTC

intelVISA wrote: post #70

jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...

Thank you for saying it out loud, I thought I was going crazy!

Re: Llama.cpp 30B runs with only 6GB of RAM now

#305

Post by freehorse » Sat, Apr 01, 2023, 1:21 PM UTC

muyuu wrote: post #297

freehorse wrote: post #193

Earlier quoted context omitted.

Same, performance of the quantised 30B model on my m1 16GB air is absolutely terrible. A couple of things I noticed on activity monitor: 1. "memory used" + "cached files" == 16GB (while swap is zero) 2. Disk reading is 500-600MB/s 3. it seems that every token is computed exactly _after every ~20GB read from disk_ which actually points that for calculating each token it actually re-reads the weights file again (instea…

Can you share the intermediate files? They're taking ages to process on my 16GB-RAM laptop

Which files are you referring to exactly?

Re: Llama.cpp 30B runs with only 6GB of RAM now

#306

Post by jart » Sat, Apr 01, 2023, 1:23 PM UTC

somesortofsystm wrote: post #281

intelVISA wrote: post #70

Earlier quoted context omitted.

Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...

> such clever use of mmap Just wanna say, that this use of mmap() is cleverly used in this context, but should be acknowledged as a widely accepted industry standard practice for getting higher performance, particularly in embedded applications but also in performance-oriented apps such as digital audio workstations, video editing systems, and so on.

Just because mmap() is commonly used doesn't mean it's commonly understood. Yes, it powers just about everything important in terms of the skeletons of our local systems. So why has the thought of using it occurred to so few people until now? Almost a whole generation has passed since things like mmap() were relegated to "the work's been done!" category of computing. People moved on to caring about things like My Browser and The Cloud where mmap() doesn't exist. Most people don't know about it. The ones who do, are reluctant to use it. Scientific computing projects are totally devoted to supporting MSVC (since you just know data scientists are secretly using those GPUs for gaming) so any thought devs may have had previously about using mmap() would have certainly triggered fears w.r.t. WIN32 before any chance to fully consider the true depth of its value would kick in. Plus data migrations are very difficult to pull off. It worked here due to the outpouring of community support, since people were blocked on this. But for a corporation with tons of cash to burn, it's a harder sell.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#307

Post by gliptic » Sat, Apr 01, 2023, 1:33 PM UTC

renewiltord wrote: post #128

diimdeep wrote: post #122

Earlier quoted context omitted.

Is the title misleading here ? 30B quantized requires 19.5 GB, not 6GB; Otherwise severe swapping to disk model original size quantized size (4-bit) 7B 13 GB 3.9 GB 13B 24 GB 7.8 GB 30B 60 GB 19.5 GB 65B 120 GB 38.5 GB

That's the size on disk, my man. When you quantize it to a smaller float size you lose precision on the weights and so the model is smaller. Then here they `mmap` the file and it only needs 6 GiB of RAM!

The size mentioned is already quantized (and to integers, not floats). mmap obviously doesn't do any quantization.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#308

Post by freehorse » Sat, Apr 01, 2023, 1:35 PM UTC

alchemist1e9 wrote: post #295

freehorse wrote: post #213

Earlier quoted context omitted.

Well in this case it does not have to do with SSDs, quite the opposite here a performance gain seems to happen by caching the file in RAM in the beginning.

That’s not my understanding. The entire point is the model can’t fit in RAM. mmap allows lazy loading from storage.

Yes but to compute a token it has to eventually read the data, either cached in RAM or from storage. There is no way that a fast SSD can compete with RAM in terms of I/O speed. To achieve any speed benefit the whole file has to be cached in RAM. This has different benefits eg threads can share memory, and the file does not have to be reread next time it is called because it is already cached in RAM, but, in final analysis, you need to have the RAM, or then you are reading from the disk, and reading 20gb for each token means you need to read 1T for a paragraph of 50 tokens. My m1, which by no means has a slow SSD, reads the file at 500-600mb/s, while a thunderbolt pci-4 enclosure reads at 700-800mb/s, even if you double that it will still take 10-20 seconds per token. To get less than 1 second per token for the 30B model one has to read there at 20gb/s. At the time we have done that, there will be even huger (v)RAMs and even larger models.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#309

Post by JBorrow » Sat, Apr 01, 2023, 1:36 PM UTC

kmeisthax wrote: post #27

detrites wrote: post #7

The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? Ok I answered my own question.

>What has everyone been doing wrong all these years So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost t…

$n00k of compute time is nothing, sorry. This is the kind of thing that academic institutions can give out for free…

Re: Llama.cpp 30B runs with only 6GB of RAM now

#310

Post by whynotminot » Sat, Apr 01, 2023, 1:36 PM UTC

pharmakom wrote: post #245

elbear wrote: post #227

Earlier quoted context omitted.

It's not the easiest syntax? It's the easiest among most popular languages. It uses the least amount of symbols, parenthesis and braces only for values. Some people don't like the significant whitespace, but that helps readability.

> It uses the least amount of symbols, parenthesis and braces only for values. is there any evidence that this makes it easier? people learn python as beginners because it has a reputation for being easy for beginners I don't see anything about the syntax that makes it inherently easier

What languages are you comparing it against?

Python is more readable than C. Way better than C++. Far simpler to reason about than Java. Maybe Typescript is on a similar level, but throwing a beginner into the JS ecosystem can be daunting. Perhaps Ruby could be argued as equally simple, but it feels like that's a dead end language these days. Golang is great, but probably not as easy to get rolling with as Python.

What else? Are you going to recommend some niche language no one hires for?