Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GGML – AI at the Edge

ggml.ai

Post reply on HN
Print view View original

221–230 of 246 posts

Re: GGML – AI at the Edge

#221

Post by wmf »

csmpltn wrote: post #212
wmf wrote: post #86

Earlier quoted context omitted.

And what do you do when every contributor to the project, including the founder, has been labeled a troublemaker?

I'm confused about the scenario you're describing here. Look, my message is simple and clear: keep the politics and drama out of it. If you partake in politics and drama, you'll be ejected from the project. I don't have the time or the energy to police or play games with people. We're here to build things, not to partake in social activism or sling crap at each other over codes of conduct, pronouns, hair color or mag…

The scenario is a mob of trolls attacking a contributor in bad faith. If you kick out the contributor, what's to stop the mob from picking off someone else?

Re: GGML – AI at the Edge

#222

Post by SparkyMcUnicorn »

gtirloni wrote: post #215
SparkyMcUnicorn wrote: post #72

Earlier quoted context omitted.

Maybe I'm wrong, but I don't think you want it fine-tuned on your data. Pretty sure you might be looking for this: https://github.com/SamurAIGPT/privateGPT Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.

> Fine-tuning is good for treating it how to act, but not great for reciting/recalling data. What underlying process makes it this way? Is it because the prompt has heavier weight?

I think your question is asking about the fundamentals of how an LLM works, which I'm not really qualified to answer. But I do have a general understanding of it all.

Fine-tuning is like having the model take a class on a certain subject. By the end of the class, it's going to have a general understanding on how to do that thing, but it's probably going to struggle when trying to quote the textbooks verbatim.

A good use-case for fine-tuning is teaching it a response style or format. If you fine-tune a model to only respond in JSON, then you no longer need to include formatting instructions in your prompt to get a JSON output.

Re: GGML – AI at the Edge

#224

Post by Art9681 »

xiphias2 wrote: post #192
georgehotz wrote: post #148

Earlier quoted context omitted.

Which low level optimizations specifically are you referring to? I'm happy with most of the abstractions. We are pushing to assembly codegen. And if you meant things like matrix accelerators, that's my next priority. We are taking more a of breadth first approach. I think ggml is more depth first and application focused. (and I think Mojo is even more breadth first)

Maybe I'd love to see Tinygrad beat GGML in its own game (4 bit LLM support on M1 Mac GPU or Tensor cores) before adding more backends / models. It's easy to debug because the generated kernels can be compared to GGML, and still gives something practical that we all can play with. At this point breadth first is a bit boring, because this way we don't know how far tinygrad is from optimal generated output.

I just deployed tinygrad thanks to this conversation and I've played with just about every local LLM client and toolchain there is. I just ran the examples as listed in the repo with absolutely zero problems and they just worked. I think their goals of prioritizing ease of use far outweighs any performance optimizations at this stage of the game. Nothing is stopping the team from integrating other projects if their performance delta is worth the pivot.

From what I see, the foundation is there for a great multimodal platform. Very excited to see where this goes.

Re: GGML – AI at the Edge

#225

Post by ariym »

bugglebeetle wrote: post #202
legendofbrando wrote: post #194

Running whisper locally on my iPhone back in December and watching perfect transcriptions pop out without sending anything to a server was a real lightbulb moment for me that set in motion a bunch of the work I’m doing now. Excited to see the new heights this unlocks!

What are you using to run Whisper locally?

whisper.cpp is optimized for Apple Silicon and is available as a Swift package

https://github.com/ggerganov/whisper.spm

Re: GGML – AI at the Edge

#227

Post by jart »

ggmlanocoward wrote: post #171
jart wrote: post #143

Earlier quoted context omitted.

Gerganov was prioritizing collaboration with 4chan who raided his GitHub to demand a change written by a transgender woman be reverted. There was so much hate speech and immaturity thrown around (words like tranny troon cucking muh model) that it's a real embarrassment (to those of us deeply want to see local models succeed) that one of the smartest guys working on the problem was taken in by all that. You can't run…

I get that a hateful mob jumped all over this widely-publicized PR and that's really, really not ok, but it doesn't make you automatically in the right. Sometimes our egos get the better of us, mistakes are made. Subsequently the only choice you have is between being someone who escalates drama and someone who defuses it. I promise you that being the latter is the better choice, even if it doesn't come with the ego-b…

Not until I'm made whole. I donated a lot of resources to the llama.cpp project. I volunteered and successfully contributed one of its most impactful features. I was rewarded with harassment and being publicly humiliated by its leader, for no reason at all. They also reneged on promises they made me. I'm owed a lot more than apology, but I haven't even received that.

Re: GGML – AI at the Edge

#228

Post by rvz »

jdonaldson wrote: post #60
rvz wrote: post #37

Earlier quoted context omitted.

> ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. > I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve t…

> I think you are setting yourself up for disappointment in the future. Why would you say that?

Never expect such promises to go your way, especially when VCs, angels, etc are able to control the project with their opaque terms sheet, which is why I am skeptical of this. Accepting VC, angel investment cash is no different to having another boss.

I am expecting such high expectations like that to end in disappointment for the 'community' since the interests will now be in the VCs to head for the exit. Their actions will speak more than what they are saying on the website.

Re: GGML – AI at the Edge

#229

Post by anaganisk »

jgrahamc wrote: post #22
throw74775 wrote: post #10

Earlier quoted context omitted.

Do you have pre-seed funding to give him?

I do.

Genuinely curious, but why didn't you? Didn't the project gain enough visibility before you noticed this post, or was there any other reason which would be helpful for others to know.

Re: GGML – AI at the Edge

#230

Post by csmpltn »

wmf wrote: post #221
csmpltn wrote: post #212

Earlier quoted context omitted.

I'm confused about the scenario you're describing here. Look, my message is simple and clear: keep the politics and drama out of it. If you partake in politics and drama, you'll be ejected from the project. I don't have the time or the energy to police or play games with people. We're here to build things, not to partake in social activism or sling crap at each other over codes of conduct, pronouns, hair color or mag…

The scenario is a mob of trolls attacking a contributor in bad faith. If you kick out the contributor, what's to stop the mob from picking off someone else?

If a bunch of random strangers (external to the project) are "attacking" your project somewhere on the internet (for example, on Twitter) - just ignore them and move on with your day. They don't have any power over your project and their opinions don't matter. Go on with your life and continue building.

If a bunch of random strangers (external to the project) are messing with your tools and workflows (stirring things up in the issue tracker, creating drama and playing games with silly Pull Requests and comments) - lock down your tools such that they can only be used by trusted members of your team. Close down and remove all bullshit conversations without spending any further time or energy on any of it. Platforms like GitHub blur the lines between "a suite of productivity tools for software development" and "a social network" - so make sure to lock down and limit the "social networks" aspects whilst optimizing for the "software development productivity" aspect. Go on with your life and continue building.

If the "attacks" happens internally within the project (between two or more members of the team) - eject all parties involved because they're clearly not here to build stuff. Go on with your life and continue building.

Your goal should be to spend your energy on building and creating, and collaborating with like-minded people on building and creating. Not on policing, moderating, or playing games with people.

Post reply on HN