Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GGML – AI at the Edge

ggml.ai

Post reply on HN
Print view View original

81–90 of 246 posts

Re: GGML – AI at the Edge

#81

Post by samwillis »

ignoramous wrote: post #77
samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

Can LLaMA be used for commerical purposes though (might limit external contributors)? I believe, FOSS alternatives like DataBricks Dolly / Together RedPajama / Eluether GPT NeoX (et al) is where the most progress is likely to be at.

Although llama.cpp started with the LLaMA model, it now supports many others.

Re: GGML – AI at the Edge

#82

Post by yukIttEft »

make3 wrote: post #76
yukIttEft wrote: post #32

Its graph execution is still full of busyloops, e.g.: https://github.com/ggerganov/llama.cpp/blob/44f906e8537fcec9... I wonder how much more efficient it would be when Taskflow lib was used instead, or even inteltbb.

does tbb work with apple Silicon?

I guess https://formulae.brew.sh/formula/tbb

Re: GGML – AI at the Edge

#83

Post by [deleted] »

isoprophlex wrote: post #75
nl wrote: post #56

Earlier quoted context omitted.

According to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.

If you go through the drudgery of integrating with all the existing channels (mail, Teams, discord, slack, traditional social media, texts, ...), such rapid finetuning speeds could enable an always up to date personality construct, modeled on you. Which is my personal holy grail towards making myself unnecessary; it'd be amazing to be doing some light gardening while the bot handles my coworkers ;)

[deleted]

Re: GGML – AI at the Edge

#84

Post by oceanplexian »

smiley1437 wrote: post #73
brucethemoose2 wrote: post #38

Earlier quoted context omitted.

We don't necessarily know... Hippo is closed source for now. Its comparable to Apache TVM's vulkan in speed on cuda, see https://github.com/mlc-ai/mlc-llm But honestly, the biggest advantage of llama.cpp for me is being able to split a model so performantly. My puny 16GB laptop can just barely , but very practically, run LLaMA 30B at almost 3 tokens/s, and do it right now. That is crazy!

>> run LLaMA 30B at almost 3 tokens/s Please tell me your config! I have an i9-10900 with 32GB of ram that only gets .7 tokens/s on a 30B model

With a single NVIDIA 3090 and the fastest inference branch of GPTQ-for-LLAMA https://github.com/qwopqwop200/GPTQ-for-LLaMa/tree/fastest-i..., I get a healthy 10-15 tokens per second on the 30B models. IMO GGML is great (And I totally use it) but it's still not as fast as running the models on GPU for now.

Re: GGML – AI at the Edge

#85

Post by gliptic »

nl wrote: post #56
moffkalast wrote: post #52

Earlier quoted context omitted.

Basically there, with what kind of VRAM and processing requirements? I doubt anyone running on a CPU can fine tune in a time frame that doesn't give them an obsolete model when they're done.

According to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.

I cannot find any such numbers in the paper. What the paper says is that MeZO converges much slower than SGD, and each step needs two forward passes.

"As a limitation, MeZO takes many steps in order to achieve strong performance."

Re: GGML – AI at the Edge

#86

Post by wmf »

csmpltn wrote: post #51
freedomben wrote: post #41

Earlier quoted context omitted.

Right. More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). TikTok was using this method for a while by ejecting both bullies and victims, and it "solved" the problem but most people see the injustice there. I'm not saying it was bad stewardship, I honestly don't know. I just agree that we shouldn't mak…

> More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). Man, nobody has time for this shit. Leave the games and the drama for the social justice warriors and the furries. People building shit ain't got time for this - ejecting trouble makers is the right way to go regardless of which "side" they're on.

And what do you do when every contributor to the project, including the founder, has been labeled a troublemaker?

Re: GGML – AI at the Edge

#87

Post by replygirl »

rafark wrote: post #57
danielbln wrote: post #44

Earlier quoted context omitted.

Not disagreeing with your points, but saying "ClosedAI" is about as clever as writing M$ for Microsoft back in the day, which is to say not very.

I think it’s ironic that M$ made ClosedAI.

Pedantic but that's not irony

Re: GGML – AI at the Edge

#88

Post by boppo1 »

pawelduda wrote: post #80

I happen to have RPi 4B with HomeAssistant. Is this something I could set up on it and integrate with HA to control it with speech, or is it overkill?

I doubt it. I'm running 4-bit 30B and 65B models with 64GB ram, a 4080 and a 7900x. The 7B models are less demanding, but even so, You'll need more than an rpi. Even then, it would be a project to get these to control something. This is more 'first baby steps' toward the edge.

Re: GGML – AI at the Edge

#90

Post by LoganDark »

csmpltn wrote: post #51
freedomben wrote: post #41

Earlier quoted context omitted.

Right. More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). TikTok was using this method for a while by ejecting both bullies and victims, and it "solved" the problem but most people see the injustice there. I'm not saying it was bad stewardship, I honestly don't know. I just agree that we shouldn't mak…

> More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). Man, nobody has time for this shit. Leave the games and the drama for the social justice warriors and the furries. People building shit ain't got time for this - ejecting trouble makers is the right way to go regardless of which "side" they're on.

> and the furries

Um, what?

Post reply on HN