GGML – AI at the Edge • Hacker News Board

Re: GGML – AI at the Edge

#51

Post by csmpltn » Tue, Jun 06, 2023, 6:19 PM UTC

freedomben wrote: post #41

PrimeMcFly wrote: post #24

Earlier quoted context omitted.

> I also like how when some controversy erupted on the project he just ejected the controversial people and moved on. Good stewardship Do you have more info on the controversy? I'm not sure ejecting developers just because of controversy is honestly good stewardship.

Right. More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). TikTok was using this method for a while by ejecting both bullies and victims, and it "solved" the problem but most people see the injustice there. I'm not saying it was bad stewardship, I honestly don't know. I just agree that we shouldn't mak…

> More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily).

Man, nobody has time for this shit. Leave the games and the drama for the social justice warriors and the furries. People building shit ain't got time for this - ejecting trouble makers is the right way to go regardless of which "side" they're on.

Re: GGML – AI at the Edge

#52

Post by moffkalast » Tue, Jun 06, 2023, 6:20 PM UTC

brucethemoose2 wrote: post #33

samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

If MeZO gets implemented, we are basically there: https://github.com/princeton-nlp/MeZO

Basically there, with what kind of VRAM and processing requirements? I doubt anyone running on a CPU can fine tune in a time frame that doesn't give them an obsolete model when they're done.

Re: GGML – AI at the Edge

#53

Post by moffkalast » Tue, Jun 06, 2023, 6:22 PM UTC

yukIttEft wrote: post #32

Its graph execution is still full of busyloops, e.g.: https://github.com/ggerganov/llama.cpp/blob/44f906e8537fcec9... I wonder how much more efficient it would be when Taskflow lib was used instead, or even inteltbb.

Someone ought to be along with a PR eventually.

Re: GGML – AI at the Edge

#54

Post by binarymax » Tue, Jun 06, 2023, 6:24 PM UTC

Havoc wrote: post #2

How common is avx on edge platforms?

svantana is correct that PCs are edge, but if you meant "mobile", then ARM in iOS and Android typically have NEON instructions for SIMD, not AVX: https://developer.arm.com/Architectures/Neon

Re: GGML – AI at the Edge

#55

Post by ggerganov » Tue, Jun 06, 2023, 6:28 PM UTC

conjecTech wrote: post #49

Congratulations! How do you plan to make money?

I'm planning to write code and have fun!

Re: GGML – AI at the Edge

#56

Post by nl » Tue, Jun 06, 2023, 6:31 PM UTC

moffkalast wrote: post #52

brucethemoose2 wrote: post #33

Earlier quoted context omitted.

If MeZO gets implemented, we are basically there: https://github.com/princeton-nlp/MeZO

Basically there, with what kind of VRAM and processing requirements? I doubt anyone running on a CPU can fine tune in a time frame that doesn't give them an obsolete model when they're done.

According to the paper it fine tunes at the speed of inference (!!)

This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.

Re: GGML – AI at the Edge

#57

Post by rafark » Tue, Jun 06, 2023, 6:32 PM UTC

danielbln wrote: post #44

behnamoh wrote: post #39

Earlier quoted context omitted.

I wonder if ClosedAI and other companies use the findings of the open source community in their products. For example, do they use QLORA to reduce the costs of training and inference? Do they quantize their models to serve non-subscribing consumers?

Not disagreeing with your points, but saying "ClosedAI" is about as clever as writing M$ for Microsoft back in the day, which is to say not very.

I think it’s ironic that M$ made ClosedAI.

Re: GGML – AI at the Edge

#58

Post by jmoss20 » Tue, Jun 06, 2023, 6:32 PM UTC

behnamoh wrote: post #39

samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

I wonder if ClosedAI and other companies use the findings of the open source community in their products. For example, do they use QLORA to reduce the costs of training and inference? Do they quantize their models to serve non-subscribing consumers?

Quantization is hardly a "finding of the open source community". (IIRC the first TPU was int8! Though the tradition is much older than that.)

Re: GGML – AI at the Edge

#59

Post by maxilevi » Tue, Jun 06, 2023, 6:35 PM UTC

ulchar wrote: post #46

rvz wrote: post #37

Earlier quoted context omitted.

> ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. > I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve t…

> The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. How exactly could they meaningfully do that? Genuine question. The issue with the OpenAI business model is that the collaboration within academia and open source circles is creating innovations that are on track to out-pace the closed source approach. Does OpenAI have the pockets to buy the open sourc…

I agree with the spirit but saying that open source is on track to outpace OpenAI in innovation is just not true. Open source models are being compared to GPT3.5, none yet even get close to GPT4 quality and they finished that last year.

Re: GGML – AI at the Edge

#60

Post by jdonaldson » Tue, Jun 06, 2023, 6:35 PM UTC

rvz wrote: post #37

samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

> ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. > I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve t…

> I think you are setting yourself up for disappointment in the future.

Why would you say that?