Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GGML – AI at the Edge

ggml.ai

Post reply on HN
Print view View original

171–180 of 246 posts

Re: GGML – AI at the Edge

#171

Post by ggmlanocoward »

jart wrote: post #143
killthebuddha wrote: post #14

Earlier quoted context omitted.

Another important detail about the ejections that I think is particularly classy is that the people he ejected are broadly considered to have world-class technical skills. In other words, he was very explicitly prioritizing collaborative potential > technical skill. Maybe a future BDFL[1]! [1] https://en.wikipedia.org/wiki/Benevolent_dictator_for_life

Gerganov was prioritizing collaboration with 4chan who raided his GitHub to demand a change written by a transgender woman be reverted. There was so much hate speech and immaturity thrown around (words like tranny troon cucking muh model) that it's a real embarrassment (to those of us deeply want to see local models succeed) that one of the smartest guys working on the problem was taken in by all that. You can't run…

I get that a hateful mob jumped all over this widely-publicized PR and that's really, really not ok, but it doesn't make you automatically in the right. Sometimes our egos get the better of us, mistakes are made. Subsequently the only choice you have is between being someone who escalates drama and someone who defuses it. I promise you that being the latter is the better choice, even if it doesn't come with the ego-boosting joy of being "right". The person who can rise above it all is the one who ends up winning respect in the long-run, but it requires acknowledging one's own fallibility in the short-term.

Re: GGML – AI at the Edge

#172

Post by jart »

jstarfish wrote: post #167
jart wrote: post #143

Earlier quoted context omitted.

Gerganov was prioritizing collaboration with 4chan who raided his GitHub to demand a change written by a transgender woman be reverted. There was so much hate speech and immaturity thrown around (words like tranny troon cucking muh model) that it's a real embarrassment (to those of us deeply want to see local models succeed) that one of the smartest guys working on the problem was taken in by all that. You can't run…

[flagged]

Are you talking about Slaren? I wrote a blog post promoting him a while back. You can read it here: https://justine.lol/mmap/ It also spotlights other unsung heroes who played important roles in helping me bring mmap() to the machine learning community. As for me being trans, the problem is that 4chan does care. The /g/ local model discussion board was consumed with discussing my gender status for more than a month. Stirring up hate is how they rally anons to venture out on raids which have damaged my professional relationships.

Re: GGML – AI at the Edge

#173

Post by menzoic »

tikkun wrote: post #121
SparkyMcUnicorn wrote: post #111

Earlier quoted context omitted.

Seems like serverless is the way to go for fast output while remaining inexpensive. e.g. https://replicate.com/stability-ai/stablelm-tuned-alpha-7b https://github.com/runpod/serverless-workers/tree/main/worke... https://modal.com/docs/guide/ex/falcon_gptq

I think that's true if you're doing minimal usage / low utilization, otherwise a dedicated instance will be cheaper.

You are correct. The pricing model guarantees this. Pay per compute vs pay for uptime (during which you could have more compute for cheaper)

Re: GGML – AI at the Edge

#174

Post by okhuman »

version_five wrote: post #166

So I'll say it: I understand why someone would do it, I'm sure they backed up a money truck, but it sucks to see this sell out. VC is going to suck all the value out and leave something that exists to funnel money to them. This has been an awesome project. I hope somebody forks it and maintains a version that isn't profit motivated.

I share the same sentiment and feel the same, its not you alone saying these things. It's starting to seem the llama.cpp project wasn't so community oriented to begin with - which in itself takes a lot of patience.

Re: GGML – AI at the Edge

#175

Post by ac29 »

s1k3s wrote: post #140

I'm out of the loop on this entire thing so call me an idiot if I get it wrong. Isn't this whole movement based on a model leak from Meta? Aren't licenses involved that prevent it from going commercial?

It wasn't a leak, LLaMa was released publicly under an open-ish license (the code is GPL, the model weights require registration and prohibit commercial use).

Re: GGML – AI at the Edge

#176

Post by mrtranscendence »

jstarfish wrote: post #99
freedomben wrote: post #41

Earlier quoted context omitted.

Right. More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). TikTok was using this method for a while by ejecting both bullies and victims, and it "solved" the problem but most people see the injustice there. I'm not saying it was bad stewardship, I honestly don't know. I just agree that we shouldn't mak…

> More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). TikTok was using this method for a while by ejecting both bullies and victims, This is SOP for American schools. It's laziness there, since education is supposed to be compulsory. They can't be bothered to investigate (and with today's hostile clima…

Right, I for one would also prefer it if people who face harassment and hate would just accept it gracefully and move on. I mean, get over yourself, amiright?

Re: GGML – AI at the Edge

#177

Post by mhh__ »

regularfry wrote: post #132
sva_ wrote: post #110

Really impressive work and I've asked this before, but is it really a good thing to have basically the whole library in a single 16k line file?

It makes syncing between llama.cpp, whisper.cpp, and ggml itself quite straightforward. I think the lesson here is that this setup has enabled some very high-speed project evolution or, at least, not got in its way. If that is surprising and you were expecting downsides, a) why; and b) where did they go?

https://git-scm.com/book/en/v2/Git-Tools-Submodules

Re: GGML – AI at the Edge

#179

Post by vgb2k18 »

isoprophlex wrote: post #75
nl wrote: post #56

Earlier quoted context omitted.

According to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.

If you go through the drudgery of integrating with all the existing channels (mail, Teams, discord, slack, traditional social media, texts, ...), such rapid finetuning speeds could enable an always up to date personality construct, modeled on you. Which is my personal holy grail towards making myself unnecessary; it'd be amazing to be doing some light gardening while the bot handles my coworkers ;)

> while the bot handles my coworkers

Or it handles their bots ;)

Re: GGML – AI at the Edge

#180

Post by addandsubtract »

pawelduda wrote: post #80

I happen to have RPi 4B with HomeAssistant. Is this something I could set up on it and integrate with HA to control it with speech, or is it overkill?

Home Assistant has been working on its own speech recognition solution. They're calling 2023 "The year of the voice": https://www.home-assistant.io/blog/2023/04/27/year-of-the-vo...
Post reply on HN