GGML – AI at the Edge • Hacker News Board

Re: GGML – AI at the Edge

#151

Post by killthebuddha » Tue, Jun 06, 2023, 10:36 PM UTC

jart wrote: post #143

killthebuddha wrote: post #14

Earlier quoted context omitted.

Another important detail about the ejections that I think is particularly classy is that the people he ejected are broadly considered to have world-class technical skills. In other words, he was very explicitly prioritizing collaborative potential > technical skill. Maybe a future BDFL[1]! [1] https://en.wikipedia.org/wiki/Benevolent_dictator_for_life

Gerganov was prioritizing collaboration with 4chan who raided his GitHub to demand a change written by a transgender woman be reverted. There was so much hate speech and immaturity thrown around (words like tranny troon cucking muh model) that it's a real embarrassment (to those of us deeply want to see local models succeed) that one of the smartest guys working on the problem was taken in by all that. You can't run…

I didn't want to not reply but I also didn't want to be swept into a potentially fraught internet argument. So, I tried to edit my comment as a middle ground, but it looks like I can't, I guess there must be a timeout. If I could edit it, I'd add the following:

"I should point out that I wasn't personally involved, haven't looked into it in detail, and that there are many different perspectives that should be considered."

Re: GGML – AI at the Edge

#152

Post by rafark » Tue, Jun 06, 2023, 10:40 PM UTC

replygirl wrote: post #87

rafark wrote: post #57

Earlier quoted context omitted.

I think it’s ironic that M$ made ClosedAI.

Pedantic but that's not irony

Why do you think so? According to the dictionary, ironic could be something paradoxical or weird.

Re: GGML – AI at the Edge

#153

Post by [deleted] » Tue, Jun 06, 2023, 10:42 PM UTC

c_o_n_v_e_x wrote: post #150

KronisLV wrote: post #104

Just today, I finished a blog post (also my latest submission, felt like could be useful to some) about how to get something like this working in a bundle of something to run models, as well as a web UI for more easy interaction - in my case that was koboldcpp, which can run GGML, both on the CPU (with OpenBLAS) and on the GPU (with CLBlast). Thanks to Hugging Face, getting Metharme, WizardLM or other models is also…

What do you mean by commodity hardware? Single server single CPU socket x86/ARM boxes? Anything that does not have a GPU?

[deleted]

Re: GGML – AI at the Edge

#154

Post by FailMore » Tue, Jun 06, 2023, 10:46 PM UTC

Remember

Re: GGML – AI at the Edge

#155

Post by FailMore » Tue, Jun 06, 2023, 10:47 PM UTC

Commenting to remember. Looks good

Re: GGML – AI at the Edge

#156

Post by CamperBob2 » Tue, Jun 06, 2023, 10:50 PM UTC

sva_ wrote: post #110

Really impressive work and I've asked this before, but is it really a good thing to have basically the whole library in a single 16k line file?

Yes. Next question

Re: GGML – AI at the Edge

#157

Post by tanseydavid » Tue, Jun 06, 2023, 10:53 PM UTC

Miraste wrote: post #118

danielbln wrote: post #44

Earlier quoted context omitted.

Not disagreeing with your points, but saying "ClosedAI" is about as clever as writing M$ for Microsoft back in the day, which is to say not very.

M$ is a silly way to call Microsoft greedy. ClosedAI is somewhat better because OpenAI's very name is a bald-faced lie, and they should be called on it. Are there more elegant ways to do that? Sure, but every time I see Altman in the news crying crocodile tears about the "dangers" of open anything I think we need all the forms of opposition we can find.

It is a colloquial spelling and they earned it, a long time ago.

Re: GGML – AI at the Edge

#158

Post by zo1 » Tue, Jun 06, 2023, 10:57 PM UTC

jart wrote: post #143

killthebuddha wrote: post #14

Earlier quoted context omitted.

Another important detail about the ejections that I think is particularly classy is that the people he ejected are broadly considered to have world-class technical skills. In other words, he was very explicitly prioritizing collaborative potential > technical skill. Maybe a future BDFL[1]! [1] https://en.wikipedia.org/wiki/Benevolent_dictator_for_life

Gerganov was prioritizing collaboration with 4chan who raided his GitHub to demand a change written by a transgender woman be reverted. There was so much hate speech and immaturity thrown around (words like tranny troon cucking muh model) that it's a real embarrassment (to those of us deeply want to see local models succeed) that one of the smartest guys working on the problem was taken in by all that. You can't run…

Really curious why you tried to rename the file format magic string to have your initials? Going from GGML (see Title of this post) to GGJT with JT being Justine Tunney? Seems quite unnecessary and bound to have rubbed a lot of people the wrong way.

Here is the official commit undoing the change:

https://github.com/ggerganov/llama.cpp/pull/711/files#diff-7...

Re: GGML – AI at the Edge

#159

Post by smiley1437 » Tue, Jun 06, 2023, 11:06 PM UTC

brucethemoose2 wrote: post #145

smiley1437 wrote: post #103

Earlier quoted context omitted.

The model I have is q4_0 I think that's 4 bit quantized I'm running in Windows using koboldcpp, maybe it's faster in Linux?

I am running linux with cublast offload, and I am using the new 3 bit quant that was just pulled in a day or two ago.

Thanks! I'll have to try the 3bit to see if that helps

Re: GGML – AI at the Edge

#160

Post by smiley1437 » Tue, Jun 06, 2023, 11:07 PM UTC

brucethemoose2 wrote: post #144

smiley1437 wrote: post #73

Earlier quoted context omitted.

>> run LLaMA 30B at almost 3 tokens/s Please tell me your config! I have an i9-10900 with 32GB of ram that only gets .7 tokens/s on a 30B model

I'n on a Ryzen 4900HS laptop with a RTX 2060. Like I said, very modest

Are you offloading layers to the RTX2060?