Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

421–430 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#421

Post by Max-Limelihood »

chaorace wrote: post #138
Miraste wrote: post #110

Earlier quoted context omitted.

Why does Python get so much flak for inefficiencies? It's really not that slow, and in ML the speed-sensitive parts are libraries in lower level languages anyway. Half of the optimization from this very post is in Python.

Python has the misfortune of competing against JS in this arena, which just so happens to have the most obsessively optimized JIT ever.

In ML? No, the best competition for Python in ML is... well, it's either C++ or Julia, depending on how you define "competition," given Python is effectively a glorified C++ interface.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#422

Post by chpatrick »

pharmakom wrote: post #288
alanfranz wrote: post #282

Earlier quoted context omitted.

What is another language working that well in a larger number of areas?

Clojure JavaScript Typescript OCaml Haskell F#

I've been using Haskell professionally for 8 years and its ecosystem is laughable compared to Python.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#423

Post by MattPalmer1086 »

pharmakom wrote: post #279
MattPalmer1086 wrote: post #266

Earlier quoted context omitted.

Anecdata, but I learned Python many years ago precisely because I found the syntax was clear. I liked the one way of doing most things philosophy, coming off working on a large C++ code base.

Being more clean than C++ doesn't prove much :)

Very true!

At the time I evaluated other languages to learn, narrowed it down to Ruby and Python, and picked Python as I felt it had a nicer syntax than Ruby. And the "one way to do things" philosophy. This was back in 2005 or so.

What other languages of that period would you say had a nicer syntax than Python?

Re: Llama.cpp 30B runs with only 6GB of RAM now

#424

Post by MacsHeadroom »

bugglebeetle wrote: post #415
MacsHeadroom wrote: post #414

Earlier quoted context omitted.

It's just the same llama.cpp repo everyone else is using. You just git clone it to your android phone in termux and then run make and you're done. https://github.com/ggerganov/llama.cpp Assuming you have the model file downloaded (you can use wget to download it) these are the instructions to install and run: pkg install git pkg install cmake pkg install build-essential git clone https://github.com/ggerganov/llama.cp…

Yeah, I’ve already been running llama.cpp locally, but not found it to perform at the level attested in the comment (30B model as a chat bot on commodity hardware). 13B runs okay, but inference appears generally too slow on to do anything useful on my MacBook. I wondered what you might be doing to get usable performance in that context.

You can change the number of threads llama.cpp uses with the -t argument. By default it only uses 4. For example, if your CPU has 16 physical cores then you can run ./main -m model.bin -t 16

16 cores would be about 4x faster than the default 4 cores. Eventually you hit memory bottlenecks. So 32 cores is not twice as fast as 13 cores unfortunately.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#425

Post by bboygravity »

int_19h wrote: post #420
remexre wrote: post #411

Earlier quoted context omitted.

Huh, I was proficient in Rust before "properly" learning C++, so maybe that accounts for it, but I didn't realize C++11 was controversial. Is it just move semantics, or are there some library things that are hard to implement?

I think what OP is saying is that decades-old systems wouldn't have C++11-compatible compilers on them.

And maybe that "C++" is now basically a bunch of different incompatible languages instead of just 1 language, depending on what "xx" is (11, 14, 17, 20, 23, etc).

It's like Python 2 vs Python 3 except even worse.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#426

Post by Max-Limelihood »

Miraste wrote: post #110
xienze wrote: post #98

Earlier quoted context omitted.

> but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? It’s several things: * Cutting-edge code, not overly concerned with optimization * Code written by scientists, who aren’t known for being the world’s greatest programmers * The obsession the research world has with using Python Not surprising that there’s a lot of…

Why does Python get so much flak for inefficiencies? It's really not that slow, and in ML the speed-sensitive parts are libraries in lower level languages anyway. Half of the optimization from this very post is in Python.

It really is that slow.

You're completely correct that the speed-sensitive parts are written in lower-level libraries, but another way to phrase that is "Python can go really fast, as long as you don't use Python." But this also means ML is effectively hamstrung into only using methods that already exist and have been coded in C++, since anything in Python would be too slow to compete.

There's lots of languages that make good tradeoffs between performance and usability. Python is not one of those languages. It is, at best, only slightly harder to use than Julia, yet orders-of-magnitude slower.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#427

Post by Max-Limelihood »

jart wrote: post #377
seydor wrote: post #320

Earlier quoted context omitted.

Optimization isn't just about speed. As you said, dropping dependencies makes it portable, embeddable, more versatile

It's also nice to not lose your mind over how crazy Python and Docker are, when all you want to do is run inference in a shell script as though it were the `cat` command. That sacred cow is going to have to come out of the temple sooner or later, and when that happens, people are going to think, wow, it's just a cow.

Have you tried Julia for this instead?

Re: Llama.cpp 30B runs with only 6GB of RAM now

#428

Post by bugglebeetle »

MacsHeadroom wrote: post #424
bugglebeetle wrote: post #415

Earlier quoted context omitted.

Yeah, I’ve already been running llama.cpp locally, but not found it to perform at the level attested in the comment (30B model as a chat bot on commodity hardware). 13B runs okay, but inference appears generally too slow on to do anything useful on my MacBook. I wondered what you might be doing to get usable performance in that context.

You can change the number of threads llama.cpp uses with the -t argument. By default it only uses 4. For example, if your CPU has 16 physical cores then you can run ./main -m model.bin -t 16 16 cores would be about 4x faster than the default 4 cores. Eventually you hit memory bottlenecks. So 32 cores is not twice as fast as 13 cores unfortunately.

Thanks! Will test that out!

Re: Llama.cpp 30B runs with only 6GB of RAM now

#429

Post by int_19h »

bboygravity wrote: post #425
int_19h wrote: post #420

Earlier quoted context omitted.

I think what OP is saying is that decades-old systems wouldn't have C++11-compatible compilers on them.

And maybe that "C++" is now basically a bunch of different incompatible languages instead of just 1 language, depending on what "xx" is (11, 14, 17, 20, 23, etc). It's like Python 2 vs Python 3 except even worse.

In my experience, C++03 code works just fine without changes on a C++11 and C++14 compilers, so no, it's not at all like Python 2/3. The few features that were ripped out were exactly the stuff that pretty much no-one was using for good reasons (e.g. throw-specifications).

Re: Llama.cpp 30B runs with only 6GB of RAM now

#430

Post by joquarky »

eru wrote: post #413
codexon wrote: post #395

Earlier quoted context omitted.

I don't see any reason for it to be less true now. Is python syntax worse than any brand new languages like rust or go? Absolutely not. It's still better. Did Google stop using it? I don't think so, but I also don't think people picked it just because Google did.

Python's syntax is ok. Btw, I wish they would take some inspiration from Haskell's syntax. Haskell also has significant whitespace, but its defined as syntactic sugar for a more traditionally syntax with curly braces and semicolons. Approximately no-one uses that curly-brace syntax, but it's good for two things: - silences the naysayers - more importantly: allows you to copy-paste code even into forms that mess up yo…

In a few years, none of this is going to matter anyway since it is likely we will be able to automatically translate everything cheaply.
Post reply on HN