Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GGML – AI at the Edge

ggml.ai

Post reply on HN
Print view View original

161–170 of 246 posts

Re: GGML – AI at the Edge

#161

Post by smiley1437 »

LoganDark wrote: post #123
smiley1437 wrote: post #103

Earlier quoted context omitted.

The model I have is q4_0 I think that's 4 bit quantized I'm running in Windows using koboldcpp, maybe it's faster in Linux?

> The model I have is q4_0 I think that's 4 bit quantized That's correct, yeah. Q4_0 should be the smallest and fastest quantized model. > I'm running in Windows using koboldcpp, maybe it's faster in Linux? Possibly. You could try using WSL to test—I think both WSL1 and WSL2 are faster than Windows (but WSL1 should be faster than WSL2).

I didn't know what WSL was, but now I do, thanks for the tip!

Re: GGML – AI at the Edge

#163

Post by jart »

zo1 wrote: post #158
jart wrote: post #143

Earlier quoted context omitted.

Gerganov was prioritizing collaboration with 4chan who raided his GitHub to demand a change written by a transgender woman be reverted. There was so much hate speech and immaturity thrown around (words like tranny troon cucking muh model) that it's a real embarrassment (to those of us deeply want to see local models succeed) that one of the smartest guys working on the problem was taken in by all that. You can't run…

Really curious why you tried to rename the file format magic string to have your initials? Going from GGML (see Title of this post) to GGJT with JT being Justine Tunney? Seems quite unnecessary and bound to have rubbed a lot of people the wrong way. Here is the official commit undoing the change: https://github.com/ggerganov/llama.cpp/pull/711/files#diff-7...

Because the previous changes to the file format were done by changing the last two initials of the magic. Someone commented on the pull request suggesting using a version field instead, which hadn't been documented, but Gerganov was so happy with the mmap() contribution that he asked me to keep the initials. So you should ask him why he wanted my initials to be there. I didn't see anyone else raise concerns until later on when the 4chan raid happened. I guess I failed to consider that folks who hate trans women would feel uncomfortable needing to mark their files with the initials of one. Here's the pull request: https://github.com/ggerganov/llama.cpp/pull/613

Re: GGML – AI at the Edge

#164

Post by SparkyMcUnicorn »

dr_dshiv wrote: post #131
SparkyMcUnicorn wrote: post #72

Earlier quoted context omitted.

Maybe I'm wrong, but I don't think you want it fine-tuned on your data. Pretty sure you might be looking for this: https://github.com/SamurAIGPT/privateGPT Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.

How does this work?

deet already gave a comprehensive answer, but I'll add that the guts of privateGPT are pretty readable and only ~200 lines of code.

Core pieces: GPT4All (LLM interface/bindings), Chroma (vector store), HuggingFaceEmbeddings (for embeddings), and Langchain to tie everything together.

https://github.com/SamurAIGPT/privateGPT/blob/main/server/pr...

Re: GGML – AI at the Edge

#165

Post by [deleted] »

SparkyMcUnicorn wrote: post #72
samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

Maybe I'm wrong, but I don't think you want it fine-tuned on your data. Pretty sure you might be looking for this: https://github.com/SamurAIGPT/privateGPT Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.

[deleted]

Re: GGML – AI at the Edge

#166

Post by version_five »

So I'll say it: I understand why someone would do it, I'm sure they backed up a money truck, but it sucks to see this sell out. VC is going to suck all the value out and leave something that exists to funnel money to them. This has been an awesome project. I hope somebody forks it and maintains a version that isn't profit motivated.

Re: GGML – AI at the Edge

#167

Post by jstarfish »

jart wrote: post #143
killthebuddha wrote: post #14

Earlier quoted context omitted.

Another important detail about the ejections that I think is particularly classy is that the people he ejected are broadly considered to have world-class technical skills. In other words, he was very explicitly prioritizing collaborative potential > technical skill. Maybe a future BDFL[1]! [1] https://en.wikipedia.org/wiki/Benevolent_dictator_for_life

Gerganov was prioritizing collaboration with 4chan who raided his GitHub to demand a change written by a transgender woman be reverted. There was so much hate speech and immaturity thrown around (words like tranny troon cucking muh model) that it's a real embarrassment (to those of us deeply want to see local models succeed) that one of the smartest guys working on the problem was taken in by all that. You can't run…

[flagged]

Re: GGML – AI at the Edge

#168

Post by nl »

sp332 wrote: post #113
nl wrote: post #56

Earlier quoted context omitted.

According to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.

It's the same memory footprint as inference. It's not that fast, and the paper mentions some optimizations that could still be done.

Yes you are right.

I completely misread that!

Re: GGML – AI at the Edge

#169

Post by nl »

rafark wrote: post #152
replygirl wrote: post #87

Earlier quoted context omitted.

Pedantic but that's not irony

Why do you think so? According to the dictionary, ironic could be something paradoxical or weird.

Well it's not paradoxical?

If one is the kind of person who writes M$ then it's pretty much expected behaviour.

Re: GGML – AI at the Edge

#170

Post by _20p0 »

zo1 wrote: post #158
jart wrote: post #143

Earlier quoted context omitted.

Gerganov was prioritizing collaboration with 4chan who raided his GitHub to demand a change written by a transgender woman be reverted. There was so much hate speech and immaturity thrown around (words like tranny troon cucking muh model) that it's a real embarrassment (to those of us deeply want to see local models succeed) that one of the smartest guys working on the problem was taken in by all that. You can't run…

Really curious why you tried to rename the file format magic string to have your initials? Going from GGML (see Title of this post) to GGJT with JT being Justine Tunney? Seems quite unnecessary and bound to have rubbed a lot of people the wrong way. Here is the official commit undoing the change: https://github.com/ggerganov/llama.cpp/pull/711/files#diff-7...

Strange comment. This doesn't sound like a legitimate criticism because of "Here is the official commit undoing the change:" being not a link to a commit to start with, and secondly being a declined pull request, and thirdly for the reality that `master` writes `ggjt` header.

Really looks like some axe-grinding here, if I'm being honest. Especially because it takes very little effort to find out what the present header is by someone who can write software.

Post reply on HN