Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GGML – AI at the Edge

ggml.ai

Post reply on HN
Print view View original

181–190 of 246 posts

Re: GGML – AI at the Edge

#181

Post by jart »

infamouscow wrote: post #129
PrimeMcFly wrote: post #112

Earlier quoted context omitted.

You've missed the point here more than I've seen anyone miss the point in a long time.

[flagged]

Just because someone can't wring your hand doesn't mean they don't deserve to influence a project. Releasing work under a license like MIT is an act of benevolence since it's giving up leverage and relying on de facto leadership. It's the hardest kind of leader to be, since de facto leaders are only followed on merit. It's not unusual at all for folks who started projects to end up caving to political pressures and getting replaced by the sorts of people you're talking about. But every now and again a Medici comes along.

Re: GGML – AI at the Edge

#182

Post by shostack »

samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

I've been trying to figure out what I might need to do in order to turn my Obsidian vault into a dataset to fine tune against. I'd invest a lot more into it now if I thought it would be a key to an AI learning about my the way it does in the movie Her.

Re: GGML – AI at the Edge

#183

Post by swyx »

okhuman wrote: post #174
version_five wrote: post #166

So I'll say it: I understand why someone would do it, I'm sure they backed up a money truck, but it sucks to see this sell out. VC is going to suck all the value out and leave something that exists to funnel money to them. This has been an awesome project. I hope somebody forks it and maintains a version that isn't profit motivated.

I share the same sentiment and feel the same, its not you alone saying these things. It's starting to seem the llama.cpp project wasn't so community oriented to begin with - which in itself takes a lot of patience.

> It's starting to seem the llama.cpp project wasn't so community oriented to begin with

what do you mean? llama.cpp has 181 contributors.

Re: GGML – AI at the Edge

#184

Post by kkielhofner »

pawelduda wrote: post #109
boppo1 wrote: post #88

Earlier quoted context omitted.

I doubt it. I'm running 4-bit 30B and 65B models with 64GB ram, a 4080 and a 7900x. The 7B models are less demanding, but even so, You'll need more than an rpi. Even then, it would be a project to get these to control something. This is more 'first baby steps' toward the edge.

The article shows example running on RPI that recognizes colour names. I could just come up with keywords that would invoke certain commands and feed them to HA, which would match them to an automation (i.e. turn off kitchen, or just kitchen ) . I think a PoC is doable, but I'm aware I could run into limitations quickly. Idk might give it a try when I'm bored. Would love voice assistant running locally but probably t…

Shameless plug, I'm the founder of Willow[0].

In short you can:

1) Run a local Willow Inference Server[1]. Supports CPU or CUDA, just about the fastest implementation of Whisper out there for "real time" speech.

2) Run local command detection on device. We pull your Home Assistant entities on setup and define basic grammar for them but any English commands (up to 400) are supported. They are recognized directly on the $50 ESP BOX device and sent to Home Assistant (or openHAB, or a REST endpoint, etc) for processing.

Whether WIS or local our performance target is 500ms from end of speech to command executed.

[0] - https://github.com/toverainc/willow

[1] - https://github.com/toverainc/willow-inference-server

Re: GGML – AI at the Edge

#185

Post by Tan-Aki »

Can anyone explain to me, in simple terms, and at a high level, what the heck am I looking at? What is this library for? What does it mean "it is used by lama.cpp and whisper.cpp"? How is it revolutionary? Thank you very much in advance!

Re: GGML – AI at the Edge

#186

Post by Tan-Aki »

Tan-Aki wrote: post #185

Can anyone explain to me, in simple terms, and at a high level, what the heck am I looking at? What is this library for? What does it mean "it is used by lama.cpp and whisper.cpp"? How is it revolutionary? Thank you very much in advance!

With maybe a tiny bit of history as well? Pretty please? : p

Re: GGML – AI at the Edge

#187

Post by 58x14 »

shostack wrote: post #182
samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

I've been trying to figure out what I might need to do in order to turn my Obsidian vault into a dataset to fine tune against. I'd invest a lot more into it now if I thought it would be a key to an AI learning about my the way it does in the movie Her.

I've been working on this for awhile now and I'd love to chat. I'll send you an email.

Re: GGML – AI at the Edge

#188

Post by fallous »

version_five wrote: post #166

So I'll say it: I understand why someone would do it, I'm sure they backed up a money truck, but it sucks to see this sell out. VC is going to suck all the value out and leave something that exists to funnel money to them. This has been an awesome project. I hope somebody forks it and maintains a version that isn't profit motivated.

I don't mind a project that is profit motivated since it incentivizes the people working on the project as well as drives them to create solutions that solve customer problems. I do, however, have a problem with VC backing due to their frequent insistence on pushing for short-term maximal share pricing rather than long-term value. Modern VC also seem to suffer from a fatal mixture of hubris and narcissism, certain of their genius and business acumen despite all evidence to the contrary.

When advising start-ups who are taking on VC funding and the inevitable "business guidance" they push on founders I like to remind them that even the best investors are only good at investing. If they were good at business, they wouldn't bother with investing in other companies but instead start companies themselves since they would own 100% and maximize their returns.

Re: GGML – AI at the Edge

#189

Post by LoganDark »

brucethemoose2 wrote: post #145
smiley1437 wrote: post #103

Earlier quoted context omitted.

The model I have is q4_0 I think that's 4 bit quantized I'm running in Windows using koboldcpp, maybe it's faster in Linux?

I am running linux with cublast offload, and I am using the new 3 bit quant that was just pulled in a day or two ago.

cuBLAS or CLBlast? There is no such thing as cublast

Re: GGML – AI at the Edge

#190

Post by legendofbrando »

58x14 wrote: post #187
shostack wrote: post #182

Earlier quoted context omitted.

I've been trying to figure out what I might need to do in order to turn my Obsidian vault into a dataset to fine tune against. I'd invest a lot more into it now if I thought it would be a key to an AI learning about my the way it does in the movie Her.

I've been working on this for awhile now and I'd love to chat. I'll send you an email.

I'm interested in this as well and have been exploring similarly. Would be super interesting to chat if you're up for it as well. Sending you an email to say hello.
Post reply on HN