GGML – AI at the Edge • Hacker News Board

Re: GGML – AI at the Edge

#31

Post by qeternity » Tue, Jun 06, 2023, 5:36 PM UTC

ggerganov wrote: post #25

qeternity wrote: post #17

Earlier quoted context omitted.

No, GGML is a CPU optimized library and quantized weight format that is closely linked to his other project llama.cpp

ggml started with focus on CPU inference, but lately we have been augmenting it with GPU support. Although still in development, it already has partial CUDA, OpenCL and Metal backend support

Hi Georgi - thanks for all the work, have been following and using since the availability of Llama base layers!

Wasn’t implying it’s CPU only, just that it started as a CPU optimized library.

Re: GGML – AI at the Edge

#32

Post by yukIttEft » Tue, Jun 06, 2023, 5:39 PM UTC

Its graph execution is still full of busyloops, e.g.:

https://github.com/ggerganov/llama.cpp/blob/44f906e8537fcec9...

I wonder how much more efficient it would be when Taskflow lib was used instead, or even inteltbb.

Re: GGML – AI at the Edge

#33

Post by brucethemoose2 » Tue, Jun 06, 2023, 5:39 PM UTC

samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

If MeZO gets implemented, we are basically there: https://github.com/princeton-nlp/MeZO

Re: GGML – AI at the Edge

#34

Post by pubby » Tue, Jun 06, 2023, 5:40 PM UTC

evanwise wrote: post #18

_20p0 wrote: post #4

This guy is damned good. I sponsored him on Github because his software is dope. I also like how when some controversy erupted on the project he just ejected the controversial people and moved on. Good stewardship. Great code. I recall something like when he first ported it and it worked on my M1 Max he hadn't even yet tested it on Apple Silicon since he didn't have the hardware. Honestly, with this and whisper, I am…

What was the controversy?

https://github.com/ggerganov/llama.cpp/pull/711

Re: GGML – AI at the Edge

#35

Post by brucethemoose2 » Tue, Jun 06, 2023, 5:42 PM UTC

sebzim4500 wrote: post #28

stri8ed wrote: post #23

Earlier quoted context omitted.

How does the quantization happen? Are the weights preprocessed before loading the model?

Yes, but to my knowledge it doesn't do any of the complicated optimization stuff that SOTA quantisation methods use. It basically is just doing a bunch of rounding. There are advantages to simplicity, after all.

Its not so simple anymore, see https://github.com/ggerganov/llama.cpp/pull/1684

Re: GGML – AI at the Edge

#36

Post by freedomben » Tue, Jun 06, 2023, 5:43 PM UTC

ggerganov wrote: post #25

qeternity wrote: post #17

Earlier quoted context omitted.

No, GGML is a CPU optimized library and quantized weight format that is closely linked to his other project llama.cpp

ggml started with focus on CPU inference, but lately we have been augmenting it with GPU support. Although still in development, it already has partial CUDA, OpenCL and Metal backend support

As a person burned by nvidia, I can't thank you enough for the OpenCL support

Re: GGML – AI at the Edge

#37

Post by rvz » Tue, Jun 06, 2023, 5:45 PM UTC

samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

> ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant

The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again.

> I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

I think you are setting yourself up for disappointment in the future.

Re: GGML – AI at the Edge

#38

Post by brucethemoose2 » Tue, Jun 06, 2023, 5:45 PM UTC

nivekney wrote: post #13

On a similar thread, how does it compare to Hippoml? Context: https://news.ycombinator.com/item?id=36168666

We don't necessarily know... Hippo is closed source for now.

Its comparable to Apache TVM's vulkan in speed on cuda, see https://github.com/mlc-ai/mlc-llm

But honestly, the biggest advantage of llama.cpp for me is being able to split a model so performantly. My puny 16GB laptop can just barely, but very practically, run LLaMA 30B at almost 3 tokens/s, and do it right now. That is crazy!

Re: GGML – AI at the Edge

#39

Post by behnamoh » Tue, Jun 06, 2023, 5:47 PM UTC

samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

I wonder if ClosedAI and other companies use the findings of the open source community in their products. For example, do they use QLORA to reduce the costs of training and inference? Do they quantize their models to serve non-subscribing consumers?

Re: GGML – AI at the Edge

#40

Post by mliker » Tue, Jun 06, 2023, 5:51 PM UTC

congrats! I was just listening to your changelog interview from months ago in which you said you were going to move on from this after you brush up the code a bit, but it seems the momentum is too great. Glad to see you carrying this amazing project(s) forward!