Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GGML – AI at the Edge

ggml.ai

Post reply on HN
Print view View original

211–220 of 246 posts

Re: GGML – AI at the Edge

#211

Post by Tan-Aki »

orost wrote: post #191
Tan-Aki wrote: post #185

Can anyone explain to me, in simple terms, and at a high level, what the heck am I looking at? What is this library for? What does it mean "it is used by lama.cpp and whisper.cpp"? How is it revolutionary? Thank you very much in advance!

ggml is a library that provides operations for running machine learning models llama.cpp is a project that uses ggml to run LLaMA, a large language model (like GPT) by Meta whisper.cpp is a project that uses ggml to run Whisper, a speech recognition model by OpenAI ggml's distinguishing feature is efficient operation on CPU. Traditionally, this sort of work is done on GPU, but GPUs with large amounts of memory are sp…

Thank you so much for your kindness Orost. Sharing really IS caring. I understand.

May good things happen to you. Peace.

Re: GGML – AI at the Edge

#212

Post by csmpltn »

wmf wrote: post #86
csmpltn wrote: post #51

Earlier quoted context omitted.

> More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). Man, nobody has time for this shit. Leave the games and the drama for the social justice warriors and the furries. People building shit ain't got time for this - ejecting trouble makers is the right way to go regardless of which "side" they're on.

And what do you do when every contributor to the project, including the founder, has been labeled a troublemaker?

I'm confused about the scenario you're describing here.

Look, my message is simple and clear: keep the politics and drama out of it. If you partake in politics and drama, you'll be ejected from the project. I don't have the time or the energy to police or play games with people. We're here to build things, not to partake in social activism or sling crap at each other over codes of conduct, pronouns, hair color or magic strings. If you're hurt - fork the project (as long as the license allows for it) and have fun playing somewhere else.

Re: GGML – AI at the Edge

#213

Post by mistercow »

detrites wrote: post #126
ignoramous wrote: post #77

Earlier quoted context omitted.

Can LLaMA be used for commerical purposes though (might limit external contributors)? I believe, FOSS alternatives like DataBricks Dolly / Together RedPajama / Eluether GPT NeoX (et al) is where the most progress is likely to be at.

May also be worth mentioning - UAE's Falcon, which apparently performs well (leads?). Falcon recently had its royalty-based commercial license modified to be fully open for free private and commercial use, via Apache 2.0: https://falconllm.tii.ae/

Hugging Face has a demo of the 40B Falcon instruct model: https://huggingface.co/blog/falcon#demo

It’s pretty good as models of that size go, although it doesn’t take a lot of playing around with it to find that there’s still a good distance between it and ChatGPT 3.5.

(I do recommend editing the instructions before playing with it though; telling a model this size that it “always tells the truth” just seems to make it overconfident and stubborn)

Re: GGML – AI at the Edge

#214

Post by henry_viii »

jart wrote: post #143
killthebuddha wrote: post #14

Earlier quoted context omitted.

Another important detail about the ejections that I think is particularly classy is that the people he ejected are broadly considered to have world-class technical skills. In other words, he was very explicitly prioritizing collaborative potential > technical skill. Maybe a future BDFL[1]! [1] https://en.wikipedia.org/wiki/Benevolent_dictator_for_life

Gerganov was prioritizing collaboration with 4chan who raided his GitHub to demand a change written by a transgender woman be reverted. There was so much hate speech and immaturity thrown around (words like tranny troon cucking muh model) that it's a real embarrassment (to those of us deeply want to see local models succeed) that one of the smartest guys working on the problem was taken in by all that. You can't run…

Just wanted to share I think you're a superstar engineer.

Also from the links you shared it looked like some users on 4chan decided to go out and harass you. If they didn't know you are a trans woman, I'm sure they would've defaulted to calling you a n***** f***** instead. But they were going to harass you nonetheless.

It was very sad to see how things developed over a small issue. I'm sure this could've gotten resolved civilly since I believe you and everyone else involved in the project had good intentions and were doing everything out of love.

Re: GGML – AI at the Edge

#215

Post by gtirloni »

SparkyMcUnicorn wrote: post #72
samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

Maybe I'm wrong, but I don't think you want it fine-tuned on your data. Pretty sure you might be looking for this: https://github.com/SamurAIGPT/privateGPT Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.

> Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.

What underlying process makes it this way? Is it because the prompt has heavier weight?

Re: GGML – AI at the Edge

#216

Post by baobabKoodaa »

SparkyMcUnicorn wrote: post #111
KronisLV wrote: post #104

Just today, I finished a blog post (also my latest submission, felt like could be useful to some) about how to get something like this working in a bundle of something to run models, as well as a web UI for more easy interaction - in my case that was koboldcpp, which can run GGML, both on the CPU (with OpenBLAS) and on the GPU (with CLBlast). Thanks to Hugging Face, getting Metharme, WizardLM or other models is also…

Seems like serverless is the way to go for fast output while remaining inexpensive. e.g. https://replicate.com/stability-ai/stablelm-tuned-alpha-7b https://github.com/runpod/serverless-workers/tree/main/worke... https://modal.com/docs/guide/ex/falcon_gptq

I think cold start times will be excessive for serverless in this use case.

Re: GGML – AI at the Edge

#217

Post by baobabKoodaa »

sva_ wrote: post #110

Really impressive work and I've asked this before, but is it really a good thing to have basically the whole library in a single 16k line file?

I guess the "clean code" crowd would like to refactor this into hundreds of files that all call each other in an incomprehensible maze, plus pulling in 20GB of dependencies from the internet during install. Because that is the way™.

Re: GGML – AI at the Edge

#219

Post by java_beyb »

iamflimflam1 wrote: post #124

I've always thought on the edge to be IoT type stuff. So running on embedded devices. But maybe that not the case?

edge brings compute close to where data is generated, cloud brings data to compute.

even processing something in a web browser is called edge. i guess due to this impression the industry is moving towards "on-device"

Re: GGML – AI at the Edge

#220

Post by SparkyMcUnicorn »

baobabKoodaa wrote: post #216
SparkyMcUnicorn wrote: post #111

Earlier quoted context omitted.

Seems like serverless is the way to go for fast output while remaining inexpensive. e.g. https://replicate.com/stability-ai/stablelm-tuned-alpha-7b https://github.com/runpod/serverless-workers/tree/main/worke... https://modal.com/docs/guide/ex/falcon_gptq

I think cold start times will be excessive for serverless in this use case.

3 second cold start is good enough for me.
Post reply on HN