Llama.cpp 30B runs with only 6GB of RAM now • Hacker News Board

Re: Llama.cpp 30B runs with only 6GB of RAM now

#361

Post by codexon » Sat, Apr 01, 2023, 6:58 PM UTC

eru wrote: post #225

codexon wrote: post #215

Earlier quoted context omitted.

There were plenty of other languages competing with python for the same niche such as perl, ruby, js, php etc... Python is superior to all of those just for syntax alone, it is easier and cleaner to both read and write.

That might be true, but it seems to generally fall under the category of 'relevant 15+ years ago', doesn't it?

I'm arguing against the point that it clearly did have the easiest syntax compared to the competition back then and not because Google was using it.

Even if it doesn't have the best syntax now (which I doubt), the tooling and libraries make it a better choice over any language that have an edge over python syntax.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#362

Post by baq » Sat, Apr 01, 2023, 7:18 PM UTC

revelio wrote: post #354

alanfranz wrote: post #336

Earlier quoted context omitted.

Any JVM language or .NET language will take more to interface with native libraries, it’s not the same. Ocaml is very niche, I feel it’s an hard sell for a general purpose language. Haskell, 3x that. JS and TS, could be. But are they so much better than Python, if better at all?

Native library interfacing isn't really Python's strong suit, interpreter plugins are quite painful to write. .NET has P/Invoke which is much nicer. JVM is getting Panama+jextract, which is the nicest yet. You can go straight from header files to pure Java bindings which don't need any extra native code at all. But it's not shipped yet :(

Python has had cffi since figuratively forever, so I’m not sure why you compare native modules to P/Invoke?

Re: Llama.cpp 30B runs with only 6GB of RAM now

#363

Post by rfoo » Sat, Apr 01, 2023, 7:19 PM UTC

wootland wrote: post #335

rfoo wrote: post #232

Earlier quoted context omitted.

The stack is very volatile and unmaintainable because it doesn't need to be maintainable. Exactly why we have unmaintainable software in other domains. During the last 10 years there are ALWAYS totally new model architecture with new operations (or in case of CV new bizarre uses of Conv). By the time you get your performant perfectly maintainable masterpiece ready it's not needed anymore. The stack optimizes for flex…

Does this mean it would be easy to move off Python all together? It seems like the problem stems from everyone using pytorch at the base layer. How realistic is it recreate those apis in another, more modern language. Coding in Rust, Go... then distributing a single binary vs. pip hell seems like it would be worth it.

Check https://pytorch.org/tutorials/advanced/cpp_frontend.html

You can easily build a standalone binary (well, it would be GiB+ if you use CUDA... but that's the cost of statically linking cu*), had you coded your model and training loop in C++.

It then happily runs everywhere as long as a NVIDIA GPU driver is available (don't need to install CUDA).

Protip: Your AI research team REALLY DON'T WANT TO DO THIS BECAUSE THEY LOVE PYTHON. Having Python, even with the dependency management shit, is a feature, not a bug.

(if you want Rust / Go and don't want to wrapping libtorch/tf then you have a lot of work to do but yeah it's possible. also there are model compiler guys [1] where the promise is model.py in model.o out you just link it with your code)

[1] https://mlc.ai

Re: Llama.cpp 30B runs with only 6GB of RAM now

#364

Post by baq » Sat, Apr 01, 2023, 7:21 PM UTC

sigi64 wrote: post #287

alanfranz wrote: post #238

Earlier quoted context omitted.

Python is the 2nd best language for everything. It doesn’t excel at anything, but anything a software can do, it can be done in Python somehow. So, a great pick when you’ve got no idea where you’re going to, when you’re prototyping, when you don’t care about performance or perfection. I agree that for large scale systems when you already know what you’re doing, Python shows its limits quite soon (and we should add th…

Python, the language with global interpret lock, Is not the 2nd best language for everything, especially in the age od multicore processors.

Python is the practical language for when you do your cpu intensive tasks outside of it as a feature, since the GIL isn’t a problem with io parallelism.

You’d do better complaining about still nascent, compared to alternatives, async support or lack of jit in the official implementation.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#365

Post by codexon » Sat, Apr 01, 2023, 7:22 PM UTC

ElFitz wrote: post #240

codexon wrote: post #215

Earlier quoted context omitted.

There were plenty of other languages competing with python for the same niche such as perl, ruby, js, php etc... Python is superior to all of those just for syntax alone, it is easier and cleaner to both read and write.

> Python is superior to all of those just for syntax alone, it is easier and cleaner to both read and write. Do you have any argument to support this, aside from personal bias?

I can make some arguments but it all boils down to personal bias and anecdotes.

The forced use of spacing to delineate blocks means you will never see a bunch of brackets eating up screen space and the common error where someone adds another line to an if statement but doesn't add braces.

Semicolons not being conventional means less screen noise and less code golf 1 liners.

The focus on imperative vs functional means you rarely ever see something like this a(b(c(d(e(f(g))))

PHP suffers greatly from poorly named standard functions on top of all of that.

Don't get me started on Ruby metaprogramming.

These are just the things I could think of off the top of my head. I do not want to spend my afternoon on this. This is just my experience looking at code for over 20 years, you either believe it or you don't. There's no scientific studies to prove that 1 syntax feature is superior.

I highly doubt that everyone chose python just because Google did. Python was a giant step in syntax compared to the competition back then, and now even if there is a new language out there right now that has a better syntax, it's not going to be better by much, and it is not going to have the tooling, libraries, or the community.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#366

Post by baq » Sat, Apr 01, 2023, 7:23 PM UTC

vkou wrote: post #359

whynotminot wrote: post #310

Earlier quoted context omitted.

What languages are you comparing it against? Python is more readable than C. Way better than C++. Far simpler to reason about than Java. Maybe Typescript is on a similar level, but throwing a beginner into the JS ecosystem can be daunting. Perhaps Ruby could be argued as equally simple, but it feels like that's a dead end language these days. Golang is great, but probably not as easy to get rolling with as Python. Wh…

> Far simpler to reason about than Java. Strong disagreement. Explicit types make reasoning about Java much easier, especially when you are in an unfamiliar codebase. Python is not quite the 'write-only' language of Perl, but it is a lot easier to write it than it is to read it.

Python is getting typescript like typing support. Slowly, yes, but way better than Java’s type system.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#367

Post by rfoo » Sat, Apr 01, 2023, 7:26 PM UTC

nunobrito wrote: post #296

rfoo wrote: post #232

Earlier quoted context omitted.

The stack is very volatile and unmaintainable because it doesn't need to be maintainable. Exactly why we have unmaintainable software in other domains. During the last 10 years there are ALWAYS totally new model architecture with new operations (or in case of CV new bizarre uses of Conv). By the time you get your performant perfectly maintainable masterpiece ready it's not needed anymore. The stack optimizes for flex…

Still a poor excuse. Had they written this in Java and things wouldn't be so difficult both on performance and maintainability. Never understood why people think that indented languages are any simpler when in fact they bring all kinds of trouble for getting things done.

There's deeplearning4j (from Theano days!), go figure why it didn't take off.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#368

Post by [deleted] » Sat, Apr 01, 2023, 7:34 PM UTC

jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

[deleted]

Re: Llama.cpp 30B runs with only 6GB of RAM now

#369

Post by jart » Sat, Apr 01, 2023, 7:36 PM UTC

shock-value wrote: post #358

jart wrote: post #325

Earlier quoted context omitted.

Please read the original link to the pull request, where I stated my change offered a 2x improvement in memory usage. You actually are able to load models 2x larger without compromising system stability, because pages are no longer being copied. That's because you previously needed 40gb of RAM to load a 20GB model, in order to ensure your file cache wasn't destroyed and need to reread from disk the next time. Now you…

It's obviously a productive change and kudos for taking it on, but much of the enthusiasm being generated here was driven by the entirely unanticipated prospect of running a model at full speed using less memory than the model's own footprint, and by the notion that inference with a dense model somehow behaved in a sparse manner at runtime. Best to be a bit more grounded here, particularly with regard to claims that…

I wanted it to be sparse. Doesn't matter if it wasn't. We're already talking about how to modify the training and evaluation to make it sparser. That's the next logical breakthrough in getting inference for larger models running on tinier machines. If you think I haven't done enough to encourage skepticism, then I'd remind you that we all share the same dream of being able to run these large language models on our own. I can't control how people feel. Especially not when the numbers reported by our tools are telling us what we want to be true.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#370

Post by VernorVintage » Sat, Apr 01, 2023, 7:55 PM UTC

fy20 wrote: post #246

politician wrote: post #24

Earlier quoted context omitted.

Roughly: OpenAIs don’t employ enough jarts. In other words, the groups of folks working on training models don’t necessarily have access to the sort of optimization engineers that are working in other areas. When all of this leaked into the open, it caused a lot of people knowledgeable in different areas to put their own expertise to the task. Some of those efforts (mmap) pay off spectacularly. Expect industry to cop…

I'd say it's probably not a priority for them right now. Of course it would save them some money if they could run their models on cheaper hardware, but they've raised $11B so I don't think that's much of a concern right now. Better to spend the efforts on pushing the model forward, which some of these optimisations may make harder.

It's a pretty big concern if you had to spend a billion on training, but 6 months later the open source community is able to replicate your training for That'd be a 10,000 fold depreciation of an asset due to a preventable oversight. Ouchies.