Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GGML – AI at the Edge

ggml.ai

Post reply on HN
Print view View original

231–240 of 246 posts

Re: GGML – AI at the Edge

#231

Post by pawelduda »

kkielhofner wrote: post #184
pawelduda wrote: post #109

Earlier quoted context omitted.

The article shows example running on RPI that recognizes colour names. I could just come up with keywords that would invoke certain commands and feed them to HA, which would match them to an automation (i.e. turn off kitchen, or just kitchen ) . I think a PoC is doable, but I'm aware I could run into limitations quickly. Idk might give it a try when I'm bored. Would love voice assistant running locally but probably t…

Shameless plug, I'm the founder of Willow[0]. In short you can: 1) Run a local Willow Inference Server[1]. Supports CPU or CUDA, just about the fastest implementation of Whisper out there for "real time" speech. 2) Run local command detection on device. We pull your Home Assistant entities on setup and define basic grammar for them but any English commands (up to 400) are supported. They are recognized directly on th…

Thank you very much, I have run into Willow before during my brief research and liked it in general, this also sounds convincing

Re: GGML – AI at the Edge

#232

Post by torginus »

rvz wrote: post #6

> Nat Friedman and Daniel Gross provided the pre-seed funding. Why? Why should VCs get involved again? They are just going to look for an exit and end up getting acquired by Apple Inc. Not again.

I wonder, if Nat Friedman, who was the CEO of GitHub until recently, would work to tie this to the Microsoft/OpenAI LLM empire?

Or am I just being paranoid?

Re: GGML – AI at the Edge

#233

Post by SkyPuncher »

SparkyMcUnicorn wrote: post #72
samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

Maybe I'm wrong, but I don't think you want it fine-tuned on your data. Pretty sure you might be looking for this: https://github.com/SamurAIGPT/privateGPT Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.

I think people want both. They want fine tuning for their style of communication and interaction. They want better rank and retrieval for rote information.

In other words, it’s like having a spouse/partner. There are certain ways that we communicate that we simply know where the other person is at or what they actually mean.

Re: GGML – AI at the Edge

#235

Post by bluepoint »

gtirloni wrote: post #215
SparkyMcUnicorn wrote: post #72

Earlier quoted context omitted.

Maybe I'm wrong, but I don't think you want it fine-tuned on your data. Pretty sure you might be looking for this: https://github.com/SamurAIGPT/privateGPT Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.

> Fine-tuning is good for treating it how to act, but not great for reciting/recalling data. What underlying process makes it this way? Is it because the prompt has heavier weight?

I just read the paper about LORA. The main idea is that you write the weights of each neural network as

W = W0 + B A

Where W0 is the trained model’s weights, which are kept fixed, and A and B are matrices but with a much much lower rank than the originals (say r = 4).

It has been shown (as mentioned in the lora paper that training for specific tasks results in low rank corrections, so this is what it is all about. I think that doing LoRa can be done locally.

[1] https://github.com/microsoft/LoRA

Re: GGML – AI at the Edge

#236

Post by SparkyMcUnicorn »

SkyPuncher wrote: post #233
SparkyMcUnicorn wrote: post #72

Earlier quoted context omitted.

Maybe I'm wrong, but I don't think you want it fine-tuned on your data. Pretty sure you might be looking for this: https://github.com/SamurAIGPT/privateGPT Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.

I think people want both. They want fine tuning for their style of communication and interaction. They want better rank and retrieval for rote information. In other words, it’s like having a spouse/partner. There are certain ways that we communicate that we simply know where the other person is at or what they actually mean.

Unless you want machine-readable responses, or some other very specific need, the benefits of a fine-tuned model aren't really going to be that much better than a prompt that asks for the style you want along with an example or two. It also raises the barrier to entry quite a bit, since the majority of computers that can run the model aren't capable of training on it.

Even if you're using OpenAI's models, gpt-3.5-turbo is going to be much better (cheaper, bigger context window, higher quality) than any of their models that can be fine-tuned.

But if you're able to fine-tune a local model, then a combination of fine-tuning and embedding is probably going to give you better results than embedding alone.

Re: GGML – AI at the Edge

#237

Post by jgrahamc »

anaganisk wrote: post #229
jgrahamc wrote: post #22

Earlier quoted context omitted.

I do.

Genuinely curious, but why didn't you? Didn't the project gain enough visibility before you noticed this post, or was there any other reason which would be helpful for others to know.

Had no idea he wanted to make it a company.

Re: GGML – AI at the Edge

#238

Post by throw74775 »

jgrahamc wrote: post #237
anaganisk wrote: post #229

Earlier quoted context omitted.

Genuinely curious, but why didn't you? Didn't the project gain enough visibility before you noticed this post, or was there any other reason which would be helpful for others to know.

Had no idea he wanted to make it a company.

That makes sense - it looked very much like a pure open source project.

I wonder if they came to him or if someone else facilitated it as opposed to it being his initiative.

Re: GGML – AI at the Edge

#239

Post by IAmNotACellist »

jart wrote: post #143
killthebuddha wrote: post #14

Earlier quoted context omitted.

Another important detail about the ejections that I think is particularly classy is that the people he ejected are broadly considered to have world-class technical skills. In other words, he was very explicitly prioritizing collaborative potential > technical skill. Maybe a future BDFL[1]! [1] https://en.wikipedia.org/wiki/Benevolent_dictator_for_life

Gerganov was prioritizing collaboration with 4chan who raided his GitHub to demand a change written by a transgender woman be reverted. There was so much hate speech and immaturity thrown around (words like tranny troon cucking muh model) that it's a real embarrassment (to those of us deeply want to see local models succeed) that one of the smartest guys working on the problem was taken in by all that. You can't run…

Liar. https://news.ycombinator.com/item?id=35455930#35458068

This user stole another user's code, closed his PR, and opened a new one where she started using words like "my work," "I'm the author," "author here," etc., and trying to cozy up to the project lead.

Gerganov figured out what was happening and actually banned her from all further contributions. The user whose code was stolen, Slaren, is still contributing.

Re: GGML – AI at the Edge

#240

Post by IAmNotACellist »

jart wrote: post #227
ggmlanocoward wrote: post #171

Earlier quoted context omitted.

I get that a hateful mob jumped all over this widely-publicized PR and that's really, really not ok, but it doesn't make you automatically in the right. Sometimes our egos get the better of us, mistakes are made. Subsequently the only choice you have is between being someone who escalates drama and someone who defuses it. I promise you that being the latter is the better choice, even if it doesn't come with the ego-b…

Not until I'm made whole. I donated a lot of resources to the llama.cpp project. I volunteered and successfully contributed one of its most impactful features. I was rewarded with harassment and being publicly humiliated by its leader, for no reason at all. They also reneged on promises they made me. I'm owed a lot more than apology, but I haven't even received that.

You didn't write that feature. Slaren did. You closed his PR and made minor changes, then gradually shifted from "our feature" to "my feature."

----

That's not the original PR. jart was working on a malloc() approach that didn't work and slaren wrote all the code actually doing mmap, which jart then rebased in a random new PR, changed to support an unnecessary version change, magic numbers, a conversion tool, and WIN32 support when that was already working in the draft PR. https://archive.ph/Uva8c

This is the original PR: https://github.com/ggerganov/llama.cpp/pull/586.

Jart's archived comments:

"my changes"

"Here's how folks in the community have been reacting to my work."

"I just wrote a change that's going to let your LLaMA models load instantly..."

https://archive.ph/PyPFZ

"I'm the author"

https://archive.ph/qFrcY

"Author here..."

"Tragedy of the commons...We're talking to a group of people who live inside scientific papers and jupyer notebooks."

"My change helps inference go faster."

"The point of my change..."

"I stated my change offered a 2x improvement in memory usage."

https://archive.ph/k34V2

"I can only take credit for a 2x recrease in RAM usage."

https://archive.ph/MBPN0

"I just wrote a change that's going to let your LLaMA models load instantly, thanks to custom malloc() and the power of mmap()"

https://archive.ph/yrMwh

slaren replied to jart on HN asking her why she was doing and saying those things, and she didn't bother to reply to him, despite replying to others in that subthread within minutes. https://archive.ph/zCfiJ

----

You didn't make whole the people you damaged or the project you attempted to harm with plagiarism and pathological levels of manipulation and lying.

Post reply on HN