Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GGML – AI at the Edge

ggml.ai

Post reply on HN
Print view View original

121–130 of 246 posts

Re: GGML – AI at the Edge

#121

Post by tikkun »

SparkyMcUnicorn wrote: post #111
KronisLV wrote: post #104

Just today, I finished a blog post (also my latest submission, felt like could be useful to some) about how to get something like this working in a bundle of something to run models, as well as a web UI for more easy interaction - in my case that was koboldcpp, which can run GGML, both on the CPU (with OpenBLAS) and on the GPU (with CLBlast). Thanks to Hugging Face, getting Metharme, WizardLM or other models is also…

Seems like serverless is the way to go for fast output while remaining inexpensive. e.g. https://replicate.com/stability-ai/stablelm-tuned-alpha-7b https://github.com/runpod/serverless-workers/tree/main/worke... https://modal.com/docs/guide/ex/falcon_gptq

I think that's true if you're doing minimal usage / low utilization, otherwise a dedicated instance will be cheaper.

Re: GGML – AI at the Edge

#122

Post by LoganDark »

oceanplexian wrote: post #84
smiley1437 wrote: post #73

Earlier quoted context omitted.

>> run LLaMA 30B at almost 3 tokens/s Please tell me your config! I have an i9-10900 with 32GB of ram that only gets .7 tokens/s on a 30B model

With a single NVIDIA 3090 and the fastest inference branch of GPTQ-for-LLAMA https://github.com/qwopqwop200/GPTQ-for-LLaMa/tree/fastest-i... , I get a healthy 10-15 tokens per second on the 30B models. IMO GGML is great (And I totally use it) but it's still not as fast as running the models on GPU for now.

> IMO GGML is great (And I totally use it) but it's still not as fast as running the models on GPU for now.

I think it was originally designed to be easily embeddable—and most importantly, native code (i.e. not Python)—rather than competitive with GPUs.

I think it's just starting to get into GPU support now, but carefully.

Re: GGML – AI at the Edge

#123

Post by LoganDark »

smiley1437 wrote: post #103
LoganDark wrote: post #79

Earlier quoted context omitted.

> Please tell me your config! I have an i9-10900 with 32GB of ram that only gets .7 tokens/s on a 30B model Have you quantized it?

The model I have is q4_0 I think that's 4 bit quantized I'm running in Windows using koboldcpp, maybe it's faster in Linux?

> The model I have is q4_0 I think that's 4 bit quantized

That's correct, yeah. Q4_0 should be the smallest and fastest quantized model.

> I'm running in Windows using koboldcpp, maybe it's faster in Linux?

Possibly. You could try using WSL to test—I think both WSL1 and WSL2 are faster than Windows (but WSL1 should be faster than WSL2).

Re: GGML – AI at the Edge

#125

Post by digitallyfree »

KronisLV wrote: post #104

Just today, I finished a blog post (also my latest submission, felt like could be useful to some) about how to get something like this working in a bundle of something to run models, as well as a web UI for more easy interaction - in my case that was koboldcpp, which can run GGML, both on the CPU (with OpenBLAS) and on the GPU (with CLBlast). Thanks to Hugging Face, getting Metharme, WizardLM or other models is also…

The fact that this is commodity hardware makes ggml extremely impressive and puts the tech in the hands of everyone. I recently reported my experience running 7B llama.cpp on a 15 year old Core 2 Quad [1] - when that machine came out it was a completely different world and I certainly never imagined how AI would look like today. This was around when the first iPhone was released and everyone began talking about how smartphones would become the next big thing. We saw what happened 15 years later...

Today with the new k-quants users are reporting that 30B models are working with 2-bit quantization on 16GB CPUs and GPUs [2]. That's enabling access to millions of consumers and the optimizations will only improve from there.

[1] https://old.reddit.com/r/LocalLLaMA/comments/13q6hu8/7b_perf...

[2] https://github.com/ggerganov/llama.cpp/pull/1684, https://old.reddit.com/r/LocalLLaMA/comments/141bdll/moneros...

Re: GGML – AI at the Edge

#126

Post by detrites »

ignoramous wrote: post #77
samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

Can LLaMA be used for commerical purposes though (might limit external contributors)? I believe, FOSS alternatives like DataBricks Dolly / Together RedPajama / Eluether GPT NeoX (et al) is where the most progress is likely to be at.

May also be worth mentioning - UAE's Falcon, which apparently performs well (leads?). Falcon recently had its royalty-based commercial license modified to be fully open for free private and commercial use, via Apache 2.0: https://falconllm.tii.ae/

Re: GGML – AI at the Edge

#127

Post by Y_Y »

iamflimflam1 wrote: post #124

I've always thought on the edge to be IoT type stuff. So running on embedded devices. But maybe that not the case?

Like any new term the (mis)usage broadens the meaning over time until it either it's widely known, it's unfashionable, or most likely; it becomes so broad as to be meaningless and hence it achieves buzzword apotheosis.

My old job title had "edge" in it, and I still don't know what it's supposed to mean, although "not cloud" is a good approximation.

Re: GGML – AI at the Edge

#128

Post by timerol »

iamflimflam1 wrote: post #124

I've always thought on the edge to be IoT type stuff. So running on embedded devices. But maybe that not the case?

"Edge computing" is a pretty vague term, and can encompass anything from a 8MHz ARM core that can barely talk compliant BLE, all the way to a multi-thousand dollar setup on something like a self-checkout machine, which may have more compute available than your average laptop. In that range are home assistants, which normally have some basic ML for wake word detection, and then send the next bit of audio to the cloud with a more advanced model for full speech-to-text (and response)

Re: GGML – AI at the Edge

#129

Post by infamouscow »

PrimeMcFly wrote: post #112
infamouscow wrote: post #100

Earlier quoted context omitted.

The code is MIT licensed. If you don't agree with the direction the project is taking you can fork it and add whatever you want. I don't understand why this is so difficult for software developers with GitHub accounts to understand.

You've missed the point here more than I've seen anyone miss the point in a long time.

[flagged]

Re: GGML – AI at the Edge

#130

Post by chaxor »

ignoramous wrote: post #77
samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

Can LLaMA be used for commerical purposes though (might limit external contributors)? I believe, FOSS alternatives like DataBricks Dolly / Together RedPajama / Eluether GPT NeoX (et al) is where the most progress is likely to be at.

Why is commercial necessary to run local models?
Post reply on HN