Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp 30B runs with only 6GB of RAM now

github.com

Post reply on HN
Print view View original

391–400 of 436 posts

Re: Llama.cpp 30B runs with only 6GB of RAM now

#391

Post by __turbobrew__ »

sheeshkebab wrote: post #21
sillysaurusx wrote: post #16

On the legal front, I’ve been working with counsel to draft a counterclaim to Meta’s DMCA against llama-dl. (GPT-4 is surprisingly capable, but I’m talking to a few attorneys: https://twitter.com/theshawwn/status/1641841064800600070?s=6... ) An anonymous HN user named L pledged $200k for llama-dl’s legal defense: https://twitter.com/theshawwn/status/1641804013791215619?s=6... This may not seem like much vs Meta, but…

All models trained on public data need to be made public. As it is their outputs are not copyrightable, it’s not a stretch to say models are public domain.

Aggregating and organizing public knowledge is a fundamentally valuable action which many companies make their business off of.

If I create a website for tracking real estate trends in my area — which is public information — should I not be able to sell that information?

Similarly if a consulting company analyzes public market macro trends are they not allowed to sell that information?

Just because the information which is being aggregated and organized is public does not necessarily mean that the output product should be in the public.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#392

Post by kazinator »

renox wrote: post #264
pharmakom wrote: post #244

Earlier quoted context omitted.

> Python is the 2nd best language for everything. Huh? Why? You can barely deploy it to Web. it doesn't scale perfoance wise you can't built robust abstractions The REPL is merely OK You can barely ship working code without containers the syntax is hard to manipulate programmatically Python has inertia but it's holding us back

You have good points but "the syntax is hard to manipulate programmatically"?? Maybe you haven't noticed but Lisp is now a tiny niche and most new languages aren't homoiconic either..

Here, we can set Lisp aside and take grandparent comment's definition of syntax to be concrete, character-level syntax.

Python concrete syntax is harder to manipulate programmatically compared to Javascript concrete syntax.

For instance, to insert one statement into another, we need to traverse the lines of that syntax and add the right amount of indentation. We can't just plant the syntax into the desired spot and be done with it.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#393

Post by datadeft »

terafo wrote: post #11
datadeft wrote: post #9

Earlier quoted context omitted.

I have predicted that LLaMA will be available on mobile phones before the end of this year. We are very close.

You mean in contained app? It can already run on a phone. GPU acceleration would be nice at this point, though.

I was thinking about the limitation by the mobile HW. Yeah GPU support would be nice.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#394

Post by datadeft »

rickrollin wrote: post #12
datadeft wrote: post #9

Earlier quoted context omitted.

I have predicted that LLaMA will be available on mobile phones before the end of this year. We are very close.

People have actually ran it on phones.

This is what my comment is about. It happened much sooner than I thought.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#395

Post by codexon »

eru wrote: post #387
codexon wrote: post #361

Earlier quoted context omitted.

I'm arguing against the point that it clearly did have the easiest syntax compared to the competition back then and not because Google was using it. Even if it doesn't have the best syntax now (which I doubt), the tooling and libraries make it a better choice over any language that have an edge over python syntax.

> I'm arguing against the point that it clearly did have the easiest syntax compared to the competition back then and not because Google was using it. Maybe, not sure? My point was that both the syntax and Google using it was more relevant 15 years ago than now. (I don't have much of an opinion on the 15+ years ago thing.)

I don't see any reason for it to be less true now.

Is python syntax worse than any brand new languages like rust or go? Absolutely not. It's still better.

Did Google stop using it? I don't think so, but I also don't think people picked it just because Google did.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#396

Post by miraculixx »

MontyCarloHall wrote: post #125
intelVISA wrote: post #70

Earlier quoted context omitted.

Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...

>how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions Probably not all that much. All of the Python numeric computing frameworks (Numpy, PyTorch, TensorFlow, etc.) are basically just wrappers for lower level C++/C/Fortran code. Unless you’re doing something boneheaded and converting framework-native tensors to Python objects, passing tens…

What a bad take!

Python is not the cause of dependency hell. Deep dependency trees are. The only way to deal with this is to use seperate environments and to carefully specify the exact requirements.

Those who claim some language would be a magical fix clearly lack experience in multiple languages.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#397

Post by miraculixx »

rfoo wrote: post #230
intelVISA wrote: post #70

Earlier quoted context omitted.

Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...

Sigh. It's not like the zero copy buzzword is going to help you during training, all your weights have to stay on GPU, you are going to sample your training data randomly and your data is on a networked storage anyway, so mmap HURTS. You'd better just O_DIRECT. Similarly, as long as you run your inference on GPU it's not like you can mmap... And I have indeed worked on inference runtimes for mobile devices and on the…

This! Thank you.

Re: Llama.cpp 30B runs with only 6GB of RAM now

#398

Post by revelio »

baq wrote: post #362
revelio wrote: post #354

Earlier quoted context omitted.

Native library interfacing isn't really Python's strong suit, interpreter plugins are quite painful to write. .NET has P/Invoke which is much nicer. JVM is getting Panama+jextract, which is the nicest yet. You can go straight from header files to pure Java bindings which don't need any extra native code at all. But it's not shipped yet :(

Python has had cffi since figuratively forever, so I’m not sure why you compare native modules to P/Invoke?

Most important libraries with native components are using C extensions, not cffi.
Post reply on HN