Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

381–390 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#381

Post by AnthonyMouse »

hunta2097 wrote: post #275
CurrentB wrote: post #235

Earlier quoted context omitted.

Yeah, I've been using python for the first time in a while to try out some of the llm stuff and I can't believe how bad the dependency hell is. It's probably particularly bad due to the pace of change in this field. But I spend an hour getting dependencies fixed every time I touch anything. 80% of the Google Collabs I find are just outright broken. I wish there were other viable non python options to try out these th…

You're using virtual environments, right? ML libraries are particularly bad, most other stuff works well. Friends don't let friends install pip into /usr/lib.

This just goes to show what a mess this is.

Suppose you have a big piece of compute hardware (e.g. at a university) which is shared by multiple users. They all want to come in and play with these models. Each one is tens to hundreds of gigabytes. Is each user supposed to have their own copy in their home directory?

Re: Llama.cpp 30B runs with only 6GB of RAM now

#382

Post by Accujack »

AnthonyMouse wrote: post #381
hunta2097 wrote: post #275

Earlier quoted context omitted.

You're using virtual environments, right? ML libraries are particularly bad, most other stuff works well. Friends don't let friends install pip into /usr/lib.

This just goes to show what a mess this is. Suppose you have a big piece of compute hardware (e.g. at a university) which is shared by multiple users. They all want to come in and play with these models. Each one is tens to hundreds of gigabytes. Is each user supposed to have their own copy in their home directory?

This is not exactly a new problem.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#384

Post by AnthonyMouse »

Accujack wrote: post #382
AnthonyMouse wrote: post #381

Earlier quoted context omitted.

This just goes to show what a mess this is. Suppose you have a big piece of compute hardware (e.g. at a university) which is shared by multiple users. They all want to come in and play with these models. Each one is tens to hundreds of gigabytes. Is each user supposed to have their own copy in their home directory?

This is not exactly a new problem.

That's kind of the point. We solved this problem decades ago. You have a system package manager that installs a system-wide copy of the package that everybody can use.

But now we encounter this broken nonsense because solved problems get unsolved by bad software.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#385

Post by MacsHeadroom »

bugglebeetle wrote: post #348
bestcoder69 wrote: post #168

Earlier quoted context omitted.

Thanks for this! I was able to integrate alpaca-30B into a slack bot & a quick tkinter GUI (coded by GPT-4 tbh) by just shelling out to `./main` in both cases, since model loading is so quick now. (I didn't even have to ask GPT-4 to code me up Python bindings to llama's c-style api!)

What’s your setup for running these? I’m not seeing performance improvements on off the shelf hardware that would allow for this.

I host a llama-13B IRC chatbot on a spare old android phone.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#386

Post by bugglebeetle »

MacsHeadroom wrote: post #385
bugglebeetle wrote: post #348

Earlier quoted context omitted.

What’s your setup for running these? I’m not seeing performance improvements on off the shelf hardware that would allow for this.

I host a llama-13B IRC chatbot on a spare old android phone.

Have a repo anywhere?

Re: Llama.cpp 30B runs with only 6GB of RAM now

#387

Post by eru »

codexon wrote: post #361
eru wrote: post #225

Earlier quoted context omitted.

That might be true, but it seems to generally fall under the category of 'relevant 15+ years ago', doesn't it?

I'm arguing against the point that it clearly did have the easiest syntax compared to the competition back then and not because Google was using it. Even if it doesn't have the best syntax now (which I doubt), the tooling and libraries make it a better choice over any language that have an edge over python syntax.

> I'm arguing against the point that it clearly did have the easiest syntax compared to the competition back then and not because Google was using it.

Maybe, not sure? My point was that both the syntax and Google using it was more relevant 15 years ago than now.

(I don't have much of an opinion on the 15+ years ago thing.)

Re: Llama.cpp 30B runs with only 6GB of RAM now

#389

Post by xsmasher »

w1nk wrote: post #284
xsmasher wrote: post #214

Earlier quoted context omitted.

Does this mean LLaMA only uses 10% of it's brain? An urban legend come to life!

No, your OP is mistaken. The model weights have to all be accessed for the forward pass. What has happened is that using mmap changes where the memory is consumed (kernel vs process) and so it was being incorrectly interpreted. There are still 30B parameters, and you'll need that times however big your floating point representation is to use the model still.

But do they all need to be accessed at the same time? If not, pages that are not being actively used can be dropped from memory until needed again.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#390

Post by opless »

remexre wrote: post #378
superkuh wrote: post #338

Earlier quoted context omitted.

Everyone is using llama.cpp because we reject the idea of giving up on system libraries like nix does. That kind of tomfoolery (at least in the desktop context) is only required when you use software projects that use libraries/languages which break forwards compatibility every 3 years. If you just write straight c++ (without c++xx, or anything like it) you can compile the code on machines from decades ago if you wan…

What's c++xx?

C++11, and greater.
Post reply on HN