Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

281–290 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#281

Post by somesortofsystm »

intelVISA wrote: post #70
jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...

> such clever use of mmap

Just wanna say, that this use of mmap() is cleverly used in this context, but should be acknowledged as a widely accepted industry standard practice for getting higher performance, particularly in embedded applications but also in performance-oriented apps such as digital audio workstations, video editing systems, and so on.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#282

Post by alanfranz »

pharmakom wrote: post #244
alanfranz wrote: post #238

Earlier quoted context omitted.

Python is the 2nd best language for everything. It doesn’t excel at anything, but anything a software can do, it can be done in Python somehow. So, a great pick when you’ve got no idea where you’re going to, when you’re prototyping, when you don’t care about performance or perfection. I agree that for large scale systems when you already know what you’re doing, Python shows its limits quite soon (and we should add th…

> Python is the 2nd best language for everything. Huh? Why? You can barely deploy it to Web. it doesn't scale perfoance wise you can't built robust abstractions The REPL is merely OK You can barely ship working code without containers the syntax is hard to manipulate programmatically Python has inertia but it's holding us back

What is another language working that well in a larger number of areas?

Re: Llama.cpp 30B runs with only 6GB of RAM now

#283

Post by somesortofsystm »

jart wrote: post #158
xmprt wrote: post #150

Earlier quoted context omitted.

This still doesn't make sense. It doesn't take a full year to do optimizations like this. Maybe a month at most if you include the investigation time. And the memory usage is $150 times the number of users which is in the thousands at least.

Tragedy of the commons. If you want to do something that benefits everyone a little bit, and you can't productize it like OpenAI's $20/month subscription, then there's no rational economic reason to do it, and you have to wait for someone like me who has an irrational love of coding. It's not a lifestyle that makes you rich, but it does help you see the opportunities to fix problems that the well-resourced folks who…

>Tragedy of the commons.

Tragedy of folks forgetting how to program.

This mmap() "trick" isn't a trick, its a standard practice for anyone who has cut their teeth on POSIX or embedded. See also mlock()/munlock() ..

Re: Llama.cpp 30B runs with only 6GB of RAM now

#284

Post by w1nk »

xsmasher wrote: post #214
heap_perms wrote: post #83

Earlier quoted context omitted.

I was thinking something similar. Turns out that you don't need all the weights for any given prompt. > LLaMA 30B appears to be a sparse model. While there's 20GB of weights, depending on your prompt I suppose only a small portion of that needs to be used at evaluation time [...] Found the answer from the author of this amazing pull request: https://github.com/ggerganov/llama.cpp/discussions/638#discu...

Does this mean LLaMA only uses 10% of it's brain? An urban legend come to life!

No, your OP is mistaken. The model weights have to all be accessed for the forward pass. What has happened is that using mmap changes where the memory is consumed (kernel vs process) and so it was being incorrectly interpreted. There are still 30B parameters, and you'll need that times however big your floating point representation is to use the model still.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#286

Post by smallerfish »

alanfranz wrote: post #238
sn_master wrote: post #196

Earlier quoted context omitted.

I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it". It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.

Python is the 2nd best language for everything. It doesn’t excel at anything, but anything a software can do, it can be done in Python somehow. So, a great pick when you’ve got no idea where you’re going to, when you’re prototyping, when you don’t care about performance or perfection. I agree that for large scale systems when you already know what you’re doing, Python shows its limits quite soon (and we should add th…

To steal from another thread, Python is the McDonald's of languages - it's ubiquitous, it doesn't take much effort, and it's really not very good.

The trope about it being the 2nd best language for everything isn't correct. It's taught in universities because it has a very short time to gratification, and the basic syntax is quite intuitive. Academics latched onto it for ML because of some excellent libraries, and it became established as a vital part of the ecosystem from there.

But it's a nightmare to support a moderate to large codebase in production, packaging continues to be a mess, and it's full of weird quirks. Great for weekend projects, but for pete's sake take a minute and port them into something more reliable before going to production with them.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#287

Post by sigi64 »

alanfranz wrote: post #238
sn_master wrote: post #196

Earlier quoted context omitted.

I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it". It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.

Python is the 2nd best language for everything. It doesn’t excel at anything, but anything a software can do, it can be done in Python somehow. So, a great pick when you’ve got no idea where you’re going to, when you’re prototyping, when you don’t care about performance or perfection. I agree that for large scale systems when you already know what you’re doing, Python shows its limits quite soon (and we should add th…

Python, the language with global interpret lock, Is not the 2nd best language for everything, especially in the age od multicore processors.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#288

Post by pharmakom »

alanfranz wrote: post #282
pharmakom wrote: post #244

Earlier quoted context omitted.

> Python is the 2nd best language for everything. Huh? Why? You can barely deploy it to Web. it doesn't scale perfoance wise you can't built robust abstractions The REPL is merely OK You can barely ship working code without containers the syntax is hard to manipulate programmatically Python has inertia but it's holding us back

What is another language working that well in a larger number of areas?

Clojure

JavaScript

Typescript

OCaml

Haskell

F#

Re: Llama.cpp 30B runs with only 6GB of RAM now

#289

Post by HarHarVeryFunny »

laichzeit0 wrote: post #259
sn_master wrote: post #196

Earlier quoted context omitted.

I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it". It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.

It’s not like there’s a gun to anyone’s head forcing them to use Python. The ecosystem (library, framework, IDEs) is what draws people to use it. If there was a superior alternative that covers the breadth of the Python ecosystem I’m pretty sure no one would have any scruples in using it. A programming language and its syntax is the least interesting or complex part when it comes to solving problems. Just rattling of…

>It’s not like there’s a gun to anyone’s head forcing them to use Python. The ecosystem (library, framework, IDEs) is what draws people to use it

Sure, but that is the gun, especially (as reflected in your examples) for machine learning. The best frameworks (PyTorch, TensorFlow, JAX) are all Python, with support for other languages being an afterthought as best.

The use of scripting languages (Python, Lua - original Torch) for ML seems to have started partly because he original users were non-developers, more from a math/stats background, and partly because an interactive REPL loop is good for a field like this that is very experimental/empirical.

Does it make sense that we're now building AGI using a scripting language? Not really, but that's where we are!

Re: Llama.cpp 30B runs with only 6GB of RAM now

#290

Post by seydor »

faitswulff wrote: post #121
abujazar wrote: post #93

I love how LLMs have got the attention of proper programmers such that the Python mess is getting cleaned up.

How so?

C has an almost infinite horizon for optimization. Python is good prototypes but we are beyond that stage now
Post reply on HN