GGML – AI at the Edge • Hacker News Board

_20p0 wrote: post #4

This guy is damned good. I sponsored him on Github because his software is dope. I also like how when some controversy erupted on the project he just ejected the controversial people and moved on. Good stewardship. Great code. I recall something like when he first ported it and it worked on my M1 Max he hadn't even yet tested it on Apple Silicon since he didn't have the hardware. Honestly, with this and whisper, I am…

Another important detail about the ejections that I think is particularly classy is that the people he ejected are broadly considered to have world-class technical skills. In other words, he was very explicitly prioritizing collaborative potential > technical skill. Maybe a future BDFL[1]!

[1] https://en.wikipedia.org/wiki/Benevolent_dictator_for_life

Re: GGML – AI at the Edge

#15

Post by world2vec » Tue, Jun 06, 2023, 5:25 PM UTC

Might be a silly question but is GGML a similar/competing library to George Hotz's tinygrad [0]?

[0] https://github.com/geohot/tinygrad

Re: GGML – AI at the Edge

#16

Post by svantana » Tue, Jun 06, 2023, 5:26 PM UTC

danieljanes wrote: post #9

Does GGML support training on the edge? We're especially interested in training support for Android+iOS

Yes - look at the file tests/test-opt.c. Unfortunately there's almost no documentation about its training/autodiff.

Re: GGML – AI at the Edge

#17

Post by qeternity » Tue, Jun 06, 2023, 5:27 PM UTC

world2vec wrote: post #15

Might be a silly question but is GGML a similar/competing library to George Hotz's tinygrad [0]? [0] https://github.com/geohot/tinygrad

No, GGML is a CPU optimized library and quantized weight format that is closely linked to his other project llama.cpp

Re: GGML – AI at the Edge

#18

Post by evanwise » Tue, Jun 06, 2023, 5:27 PM UTC

_20p0 wrote: post #4

This guy is damned good. I sponsored him on Github because his software is dope. I also like how when some controversy erupted on the project he just ejected the controversial people and moved on. Good stewardship. Great code. I recall something like when he first ported it and it worked on my M1 Max he hadn't even yet tested it on Apple Silicon since he didn't have the hardware. Honestly, with this and whisper, I am…

What was the controversy?

Re: GGML – AI at the Edge

#19

Post by samwillis » Tue, Jun 06, 2023, 5:28 PM UTC

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data.

I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

Re: GGML – AI at the Edge

#20

Post by nchudleigh » Tue, Jun 06, 2023, 5:28 PM UTC

_20p0 wrote: post #4

This guy is damned good. I sponsored him on Github because his software is dope. I also like how when some controversy erupted on the project he just ejected the controversial people and moved on. Good stewardship. Great code. I recall something like when he first ported it and it worked on my M1 Max he hadn't even yet tested it on Apple Silicon since he didn't have the hardware. Honestly, with this and whisper, I am…

he has been amazing to watch and has even helped me out with my app that uses his whisper.cpp project (https://superwhisper.com)

Excited to see how his venture goes!