Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

351–360 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#351

Post by sireat »

pharmakom wrote: post #277
l33tman wrote: post #262

Earlier quoted context omitted.

Well for starters, web deployment isn't "everything". Python is the de-facto go-to language for research or general prototyping, where not everyone is a programming wiz keeping track of the latest trendy new compiled language. Not everyone can compile stuff even.. :) Having said that, I've deployed two large Django projects on the web with tons of customers and it runs and scales just fine, and it's a DREAM to mainta…

I was talking about running in the Web browser. it's not everything, but it's an important part of everything in my book.

https://github.com/pyodide/pyodide is pretty amazing for running Python client side in the browser.

You could run notebooks entirely client side https://jupyterlite.readthedocs.io/en/latest/

The startup is slow but otherwise it is pretty functional.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#353

Post by kccqzy »

losteric wrote: post #183
kccqzy wrote: post #155

I might be missing something but I actually couldn't reproduce. I purposefully chose a computer with 16GiB RAM to run the 30B model. Performance was extremely slow, and the process was clearly not CPU-limited, unlike when it's running the 13B model. It's clearly swapping a lot.

Are the weights on NVME? Old SSD? HDD?

In my case they are on a SATA SSD.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#354

Post by revelio »

alanfranz wrote: post #336
pharmakom wrote: post #288

Earlier quoted context omitted.

Clojure JavaScript Typescript OCaml Haskell F#

Any JVM language or .NET language will take more to interface with native libraries, it’s not the same. Ocaml is very niche, I feel it’s an hard sell for a general purpose language. Haskell, 3x that. JS and TS, could be. But are they so much better than Python, if better at all?

Native library interfacing isn't really Python's strong suit, interpreter plugins are quite painful to write.

.NET has P/Invoke which is much nicer.

JVM is getting Panama+jextract, which is the nicest yet. You can go straight from header files to pure Java bindings which don't need any extra native code at all. But it's not shipped yet :(

Re: Llama.cpp 30B runs with only 6GB of RAM now

#355

Post by vkou »

bboygravity wrote: post #252
CurrentB wrote: post #235

Earlier quoted context omitted.

Yeah, I've been using python for the first time in a while to try out some of the llm stuff and I can't believe how bad the dependency hell is. It's probably particularly bad due to the pace of change in this field. But I spend an hour getting dependencies fixed every time I touch anything. 80% of the Google Collabs I find are just outright broken. I wish there were other viable non python options to try out these th…

No idea what a Google Collab is, but does the code come with an environment or at least a specifications of which packages and versions to use (requirements.txt)? It sounds unnecessarily weird to me that people would share Python code that simply doesn't work out at all out of the box.

> No idea what a Google Collab is

It's ~equivalent to a Jupyter notebook.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#356

Post by alchemist1e9 »

freehorse wrote: post #308
alchemist1e9 wrote: post #295

Earlier quoted context omitted.

That’s not my understanding. The entire point is the model can’t fit in RAM. mmap allows lazy loading from storage.

Yes but to compute a token it has to eventually read the data, either cached in RAM or from storage. There is no way that a fast SSD can compete with RAM in terms of I/O speed. To achieve any speed benefit the whole file has to be cached in RAM. This has different benefits eg threads can share memory, and the file does not have to be reread next time it is called because it is already cached in RAM, but, in final ana…

PCIe 5 NVMe drives can do 11+ GBps so at least 2x your numbers. We seem to be talking past each other because the point of the change is to run inference on a cpu for an LLM with a larger weight size than the host RAM can fit.

It looks to me that if I was planning on building a new machine capable of LLM inference it’s going to be possible using commodity gamer components and if lazy weights is viable, then such a machine with multiple PCIe 5 nvme drives in a raid 0 can potentially almost reach memory bandwidth.

On my list of to investigate next is in regards to inference with GPUs, could somehow multiple smaller GPUs be used with a technique similar to the OP post.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#357

Post by [deleted] »

superkuh wrote: post #338
pablo1107 wrote: post #302

Earlier quoted context omitted.

Maybe using Nix it's a better experience for creating such an environment where you depending also on system utilities.

Everyone is using llama.cpp because we reject the idea of giving up on system libraries like nix does. That kind of tomfoolery (at least in the desktop context) is only required when you use software projects that use libraries/languages which break forwards compatibility every 3 years. If you just write straight c++ (without c++xx, or anything like it) you can compile the code on machines from decades ago if you wan…

[deleted]

Re: Llama.cpp 30B runs with only 6GB of RAM now

#358

Post by shock-value »

jart wrote: post #325
shock-value wrote: post #303

Earlier quoted context omitted.

It appears that this was just a misreading of how memory usage was being reported and there was actually no improvement here. At least nothing so sensational as being able to run a larger-than-RAM model without swapping from disk on every iteration.

Please read the original link to the pull request, where I stated my change offered a 2x improvement in memory usage. You actually are able to load models 2x larger without compromising system stability, because pages are no longer being copied. That's because you previously needed 40gb of RAM to load a 20GB model, in order to ensure your file cache wasn't destroyed and need to reread from disk the next time. Now you…

It's obviously a productive change and kudos for taking it on, but much of the enthusiasm being generated here was driven by the entirely unanticipated prospect of running a model at full speed using less memory than the model's own footprint, and by the notion that inference with a dense model somehow behaved in a sparse manner at runtime. Best to be a bit more grounded here, particularly with regard to claims that defy common understanding.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#359

Post by vkou »

whynotminot wrote: post #310
pharmakom wrote: post #245

Earlier quoted context omitted.

> It uses the least amount of symbols, parenthesis and braces only for values. is there any evidence that this makes it easier? people learn python as beginners because it has a reputation for being easy for beginners I don't see anything about the syntax that makes it inherently easier

What languages are you comparing it against? Python is more readable than C. Way better than C++. Far simpler to reason about than Java. Maybe Typescript is on a similar level, but throwing a beginner into the JS ecosystem can be daunting. Perhaps Ruby could be argued as equally simple, but it feels like that's a dead end language these days. Golang is great, but probably not as easy to get rolling with as Python. Wh…

> Far simpler to reason about than Java.

Strong disagreement. Explicit types make reasoning about Java much easier, especially when you are in an unfamiliar codebase.

Python is not quite the 'write-only' language of Perl, but it is a lot easier to write it than it is to read it.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#360

Post by AnthonyMouse »

jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

Gosh, thank you for getting to this before I did. The first thing I said when I saw it loading tens of GB from the disk on each run is, is there some reason they're not using mmap?

This isn't just a matter of making the 30B model run in 6GB or whatever. You can now run the largest model, without heavy quantization, and let the OS figure it out. It won't be as fast as having "enough" memory, but it will run.

In theory you could always have done this with swap, but swap is even slower because evictions have to be written back to swap (and wear out your SSD if your swap isn't on glacially slow spinning rust) instead of just discarded because the OS knows where to read it back from the filesystem.

This should also make it much more efficient to run multiple instances at once because they can share the block cache.

(I wonder if anybody has done this with Stable Diffusion etc.)

Post reply on HN