GGML – AI at the Edge • Hacker News Board

Re: GGML – AI at the Edge

#21

Post by sroussey » Tue, Jun 06, 2023, 5:29 PM UTC

rvz wrote: post #6

> Nat Friedman and Daniel Gross provided the pre-seed funding. Why? Why should VCs get involved again? They are just going to look for an exit and end up getting acquired by Apple Inc. Not again.

Daniel Gross is a good guy, a yes his company did get acquired by apple a while back, but he loves to foster really dope stuff by amazing people, and ggml certainly fits the bill. And this looks like an Angel investment, not a VC one if that makes any difference to you.

Re: GGML – AI at the Edge

#22

Post by jgrahamc » Tue, Jun 06, 2023, 5:29 PM UTC

throw74775 wrote: post #10

rvz wrote: post #6

> Nat Friedman and Daniel Gross provided the pre-seed funding. Why? Why should VCs get involved again? They are just going to look for an exit and end up getting acquired by Apple Inc. Not again.

Do you have pre-seed funding to give him?

I do.

Re: GGML – AI at the Edge

#23

Post by stri8ed » Tue, Jun 06, 2023, 5:30 PM UTC

qeternity wrote: post #17

world2vec wrote: post #15

Might be a silly question but is GGML a similar/competing library to George Hotz's tinygrad [0]? [0] https://github.com/geohot/tinygrad

No, GGML is a CPU optimized library and quantized weight format that is closely linked to his other project llama.cpp

How does the quantization happen? Are the weights preprocessed before loading the model?

Re: GGML – AI at the Edge

#24

Post by PrimeMcFly » Tue, Jun 06, 2023, 5:32 PM UTC

_20p0 wrote: post #4

This guy is damned good. I sponsored him on Github because his software is dope. I also like how when some controversy erupted on the project he just ejected the controversial people and moved on. Good stewardship. Great code. I recall something like when he first ported it and it worked on my M1 Max he hadn't even yet tested it on Apple Silicon since he didn't have the hardware. Honestly, with this and whisper, I am…

> I also like how when some controversy erupted on the project he just ejected the controversial people and moved on. Good stewardship

Do you have more info on the controversy? I'm not sure ejecting developers just because of controversy is honestly good stewardship.

Re: GGML – AI at the Edge

#25

Post by ggerganov » Tue, Jun 06, 2023, 5:32 PM UTC

qeternity wrote: post #17

world2vec wrote: post #15

Might be a silly question but is GGML a similar/competing library to George Hotz's tinygrad [0]? [0] https://github.com/geohot/tinygrad

No, GGML is a CPU optimized library and quantized weight format that is closely linked to his other project llama.cpp

ggml started with focus on CPU inference, but lately we have been augmenting it with GPU support. Although still in development, it already has partial CUDA, OpenCL and Metal backend support

Re: GGML – AI at the Edge

#26

Post by okhuman » Tue, Jun 06, 2023, 5:32 PM UTC

rvz wrote: post #6

> Nat Friedman and Daniel Gross provided the pre-seed funding. Why? Why should VCs get involved again? They are just going to look for an exit and end up getting acquired by Apple Inc. Not again.

+1. VC involvement in projects like these always pivot the team away from the core competency of what you'd expect them to deliver - into some commercialization aspect that convert only a tiny fraction of the community yet take up 60%+ of the core developer team's time.

I don't know why project founders head this way...as the track records of leaders who do this end up disappointing the involved community at some point. Look to matt klein + cloud native computing foundation at envoy for a somewhat decent model of how to do this better.

We continue down the Open Core model yet it continues to fail communities.

Re: GGML – AI at the Edge

#27

Post by svantana » Tue, Jun 06, 2023, 5:32 PM UTC

Havoc wrote: post #2

How common is avx on edge platforms?

Edge just means that the computing is done close to the I/O data, so that includes PCs and such.

Re: GGML – AI at the Edge

#28

Post by sebzim4500 » Tue, Jun 06, 2023, 5:35 PM UTC

stri8ed wrote: post #23

qeternity wrote: post #17

Earlier quoted context omitted.

No, GGML is a CPU optimized library and quantized weight format that is closely linked to his other project llama.cpp

How does the quantization happen? Are the weights preprocessed before loading the model?

Yes, but to my knowledge it doesn't do any of the complicated optimization stuff that SOTA quantisation methods use. It basically is just doing a bunch of rounding.

There are advantages to simplicity, after all.

Re: GGML – AI at the Edge

#29

Post by kretaceous » Tue, Jun 06, 2023, 5:35 PM UTC

Georgi's Twitter announcement: https://twitter.com/ggerganov/status/1666120568993730561

Re: GGML – AI at the Edge

#30

Post by ggerganov » Tue, Jun 06, 2023, 5:36 PM UTC

stri8ed wrote: post #23

qeternity wrote: post #17

Earlier quoted context omitted.

No, GGML is a CPU optimized library and quantized weight format that is closely linked to his other project llama.cpp

How does the quantization happen? Are the weights preprocessed before loading the model?

The weights are preprocessed into integer quants combined with scaling factors in various configurations (4, 5, 8-bits and recently more exotic 2, 3 and 6-bit quants). At runtime, we use efficient SIMD implementations to perform the matrix multiplication at integer level, carefully optimizing for both compute and memory bandwidth. Similar strategies are applied when running GPU inference - using custom kernels for fast Matrix x Vector multiplications