Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

341–350 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#341

Post by DoctorOetker »

barbariangrunge wrote: post #333
DoctorOetker wrote: post #323

Is there a reason Llama is getting so much attention compared to say T5 11B? Not sure how neutral or what benchmarks are used on the following link, but T5 seems to sit a lot higher on this leaderboard? https://accubits.com/large-language-models-leaderboard/

Is llama open source? I heard it was pirated from Facebook

I did not claim llama was open source, but I see the url I posted insinuates that (probably for a contorted meaning of open source, as in source available for approved academics).

Anyway, T5 being available for download from Huggingface only makes my question more pertinent...

Re: Llama.cpp 30B runs with only 6GB of RAM now

#342

Post by sroussey »

wootland wrote: post #335
rfoo wrote: post #232

Earlier quoted context omitted.

The stack is very volatile and unmaintainable because it doesn't need to be maintainable. Exactly why we have unmaintainable software in other domains. During the last 10 years there are ALWAYS totally new model architecture with new operations (or in case of CV new bizarre uses of Conv). By the time you get your performant perfectly maintainable masterpiece ready it's not needed anymore. The stack optimizes for flex…

Does this mean it would be easy to move off Python all together? It seems like the problem stems from everyone using pytorch at the base layer. How realistic is it recreate those apis in another, more modern language. Coding in Rust, Go... then distributing a single binary vs. pip hell seems like it would be worth it.

Go would be interesting for the reason you could send an executable.

I’d love for JS/TS to dominate as well. Use ‘bun bun’ to send an executable if need be, but also use in in web backends.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#344

Post by kmeisthax »

breck wrote: post #329
chii wrote: post #178

Earlier quoted context omitted.

> Exact training code isn't always available, and OpenAI has even gone so far as to refuse to say anything about GPT-4's architecture or training set to prevent open replication. this is why i think the patent and copyright system is a failure. The idea that having laws protecting information like this would advance the progress of science. It doesn't, because look how an illegally leaked model gets much more advance…

> The laws protecting IP merely gives a moat to incumbents. Yes. These laws are bad. We could fix this with a 2 line change: Section 1. Article I, Section 8, Clause 8 of this Constitution is hereby repealed. Section 2. Congress shall make no law abridging the right of the people to publish information.

Abolishing the copyright clause would not solve this problem because OpenAI is not leveraging copyright or patents. They're just not releasing anything.

To fix this, you'd need to ban trade secrecy entirely. As in, if you have some kind of invention or creative work you must publish sufficient information to replicate it "in a timely manner". This would be one of those absolutely insane schemes that only a villain in an Ayn Rand book would come up with.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#345

Post by sp332 »

hackernewds wrote: post #173
sp332 wrote: post #120

Earlier quoted context omitted.

In March 2014, Tunney petitioned the US government on We the People to hold a referendum asking for support to retire all government employees with full pensions, transfer administrative authority to the technology industry, and appoint the executive chairman of Google Eric Schmidt as CEO of America https://en.m.wikipedia.org/wiki/Justine_Tunney

what's your point? also interestingly JART is in the thread here, so they might have read your comment :)

I'm pretty sure she knows already?

Re: Llama.cpp 30B runs with only 6GB of RAM now

#346

Post by colinsane »

rfoo wrote: post #232
oceanplexian wrote: post #205

Earlier quoted context omitted.

It’s not that the performance is the issue, it’s that it’s unmaintainable and prone to break. Exceptions aren’t handled right, dependencies are a disaster (Proprietary NVIDIA drivers+CUDA+PyTorch+ the various versions of stuff are a complete disaster) This leads to all sorts of bugs and breaking changes that are cool in an academic or hobbyist setting but a total headache on a large production system.

The stack is very volatile and unmaintainable because it doesn't need to be maintainable. Exactly why we have unmaintainable software in other domains. During the last 10 years there are ALWAYS totally new model architecture with new operations (or in case of CV new bizarre uses of Conv). By the time you get your performant perfectly maintainable masterpiece ready it's not needed anymore. The stack optimizes for flex…

> The stack optimizes for flexibility and iteration speed naturally

“unmaintainable” (as in “i’m spending an hour each day sorting out which dep update broke my project”) usually gets in the way of the former point.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#347

Post by xxpor »

pharmakom wrote: post #288
alanfranz wrote: post #282

Earlier quoted context omitted.

What is another language working that well in a larger number of areas?

Clojure JavaScript Typescript OCaml Haskell F#

Outside of typescript, this feels like a response from a decade ago, when Python was still mired in the 2 vs 3 problem.

What's happened to the popularity of all of these languages since 2010? Outside of JS/TS, absolutely nothing. If anything, they've lost mindshare.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#348

Post by bugglebeetle »

bestcoder69 wrote: post #168
jart wrote: post #8

Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…

Thanks for this! I was able to integrate alpaca-30B into a slack bot & a quick tkinter GUI (coded by GPT-4 tbh) by just shelling out to `./main` in both cases, since model loading is so quick now. (I didn't even have to ask GPT-4 to code me up Python bindings to llama's c-style api!)

What’s your setup for running these? I’m not seeing performance improvements on off the shelf hardware that would allow for this.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#349

Post by sroussey »

rfoo wrote: post #230
intelVISA wrote: post #70

Earlier quoted context omitted.

Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...

Sigh. It's not like the zero copy buzzword is going to help you during training, all your weights have to stay on GPU, you are going to sample your training data randomly and your data is on a networked storage anyway, so mmap HURTS. You'd better just O_DIRECT. Similarly, as long as you run your inference on GPU it's not like you can mmap... And I have indeed worked on inference runtimes for mobile devices and on the…

On a Mac, mmap definitely works for the GPU since it’s all the same unified memory.
Post reply on HN