Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GGML – AI at the Edge

ggml.ai

Post reply on HN
Print view View original

61–70 of 246 posts

Re: GGML – AI at the Edge

#61

Post by okhuman »

wmf wrote: post #47
okhuman wrote: post #45

The establishment of ggml.ai a company focusing ggml and llama.cpp, the most innovative and exciting platform to come for local LLMs, on a Open Core model is just laziness. Just because you can (and have the connections), doesn't mean you should. It's a sad state of OSS when the best most brightest developers/founders reach for antiquated models. Maybe we take up a new rules in OSS communities that say you must relea…

This looks off-topic since GGML has not announced anything about open core and their software is already MIT. More generally, if you want to take away somebody's business model you need to provide one that works. It isn't easy.

Agreed with you 100% - its not easy. Sometimes I just wish someone as talented as Georgi would innovate not just on the core tech side but bring that same tenancy to the licensing side, in a way that aligns incentives better and tries out something new. And that the community would have his back if some new approach failed, no matter what.

Re: GGML – AI at the Edge

#62

Post by freedomben »

csmpltn wrote: post #51
freedomben wrote: post #41

Earlier quoted context omitted.

Right. More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). TikTok was using this method for a while by ejecting both bullies and victims, and it "solved" the problem but most people see the injustice there. I'm not saying it was bad stewardship, I honestly don't know. I just agree that we shouldn't mak…

> More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). Man, nobody has time for this shit. Leave the games and the drama for the social justice warriors and the furries. People building shit ain't got time for this - ejecting trouble makers is the right way to go regardless of which "side" they're on.

I would agree that there needs to be a balance because wasting time babysitting adults is dumb, but what if one person is a good and loved contributor, and the other is a social justice warrior new to the project that is picking fights with the contributor? Your philosophy makes not only bad stewardship but an injustice. I'm not suggesting this is the only scenario, just merely a hypothetical that I think illustrates my position.

Re: GGML – AI at the Edge

#63

Post by moffkalast »

nl wrote: post #56
moffkalast wrote: post #52

Earlier quoted context omitted.

Basically there, with what kind of VRAM and processing requirements? I doubt anyone running on a CPU can fine tune in a time frame that doesn't give them an obsolete model when they're done.

According to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.

Wow if that's true then it's genuinely a complete gamechanger for LLMs as a whole. You probably mean more like 0.3s per token, not per example, but that's still more like 1 or two minutes per training case, not like a day for 4 cases like it is now.

Re: GGML – AI at the Edge

#64

Post by aryamaan »

Could someone at high level talk more about how one starts contributing to this kind of problems.

For the people who build solutions for data handling— ranging from crud to building highly scalable solutions— these things are alien concepts. (Or maybe I am just talking about it myself)

Re: GGML – AI at the Edge

#65

Post by valval »

nl wrote: post #56
moffkalast wrote: post #52

Earlier quoted context omitted.

Basically there, with what kind of VRAM and processing requirements? I doubt anyone running on a CPU can fine tune in a time frame that doesn't give them an obsolete model when they're done.

According to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.

I think more importantly, what would the fine tuning routine look like? It's a non-trivial task to dump all of your personal data into any LLM architecture.

Re: GGML – AI at the Edge

#66

Post by f_devd »

nl wrote: post #56
moffkalast wrote: post #52

Earlier quoted context omitted.

Basically there, with what kind of VRAM and processing requirements? I doubt anyone running on a CPU can fine tune in a time frame that doesn't give them an obsolete model when they're done.

According to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.

MeZO assumes a smooth parameter space, so you probably won't be able to do it with INT4/8 quantization, probably needs fp8 or smoother.

Re: GGML – AI at the Edge

#67

Post by Havoc »

binarymax wrote: post #54
Havoc wrote: post #2

How common is avx on edge platforms?

svantana is correct that PCs are edge, but if you meant "mobile", then ARM in iOS and Android typically have NEON instructions for SIMD, not AVX: https://developer.arm.com/Architectures/Neon

I was thinking more edge in the distributed serverless sense, but I guess for this type of use the compute part is slow not the latency so question doesn't make much sense in hindsight

Re: GGML – AI at the Edge

#68

Post by yyyk »

ulchar wrote: post #46
rvz wrote: post #37

Earlier quoted context omitted.

> ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. > I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve t…

> The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. How exactly could they meaningfully do that? Genuine question. The issue with the OpenAI business model is that the collaboration within academia and open source circles is creating innovations that are on track to out-pace the closed source approach. Does OpenAI have the pockets to buy the open sourc…

I've been going on and on about this in HN: Open source can win this fight, but I think OSS is overconfident. We need to be clear there are serious challenges ahead - ClosedAI and other corporations also have a plan, a plan that has good chances unless properly countered:

A) Embed OpenAI (etc.) API everywhere. Make embedding easy and trivial. First to gain a small API/install moat (user/dev: 'why install OSS model when OpenAI is already available with an OS API?'). If it's easy to use OpenAI but not open source they have an advantage. Second to gain brand. But more importantly:

B) Gain a technical moat by having a permanent data advantage using the existing install base (see above). Retune constantly to keep it.

C) Combine with existing propriety data stores to increase local data advantage (e.g. easy access for all your Office 365/GSuite documents, while OSS gets the scary permission prompts).

D) Combine with existing propriety moats to mutually reinforce.

E) Use selective copyright enforcement to increase data advantage.

F) Lobby legislators for limits that make competition (open or closed source) way harder.

TL;DR: OSS is probably catching up on algorithms. When it comes to good data and good integrations OSS is far behind and not yet catching up. It's been argued that OpenAI's entire performance advantage is due to having better data alone, and they intend to keep that advantage.

Re: GGML – AI at the Edge

#69

Post by okhuman »

wmf wrote: post #48
okhuman wrote: post #26

Earlier quoted context omitted.

+1. VC involvement in projects like these always pivot the team away from the core competency of what you'd expect them to deliver - into some commercialization aspect that convert only a tiny fraction of the community yet take up 60%+ of the core developer team's time. I don't know why project founders head this way...as the track records of leaders who do this end up disappointing the involved community at some poi…

Developers shouldn't be unpaid slaves to the community.

You're right. I just wish this decision was taken to the community, we could have all came together to help and supported during these difficult/transitional times. :( Maybe this decision was rushed or is money related, who knows the actual circumstances.

Here's the Matt K article https://mattklein123.dev/2021/09/14/5-years-envoy-oss/

Post reply on HN